Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Still Lags
Published: 2011-04-07T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The F-35 Lightning II, also known as the Joint Strike Fighter (JSF), is the Department of Defense's (DOD) most costly and ambitious aircraft acquisition, seeking to simultaneously develop and field three aircraft variants for the Air Force, Navy, Marine Corps, and eight international partners. The JSF is critical for recapitalizing tactical air forces and will require a long-term commitment to very large annual funding outlays. The current estimated investment is $382 billion to develop and procure 2,457 aircraft. This report, prepared in response to a congressional mandate in the National Defense Authorization Act for Fiscal Year 2010, discusses (1) program cost and schedule changes and their implications on affordability; (2) progress made during 2010; (3) design and manufacturing maturity; and (4) test plans and progress. GAO's work included analyses of a wide range of program documents and interviews with defense and contractor officials.
section_title: What GAO Found
paragraphs: DOD continues to substantially restructure the JSF program, taking positive actions that should lead to more achievable and predictable outcomes. Restructuring has consequences--higher up-front development costs, fewer aircraft in the near term, training delays, and extended times for testing and delivering capabilities to warfighters. Total development funding is now $56.4 billion to complete in 2018, a 26 percent increase in cost and a 5-year slip in schedule compared to the current baseline. DOD also reduced procurement quantities by 246 aircraft through 2016, but has not calculated the net effects of restructuring on total procurement costs nor approved a new baseline. Affordability for the U.S. and partners is challenged by a near doubling in average unit prices since program start and higher estimated life-cycle costs. Going forward, the JSF requires unprecedented funding levels in a period of more austere defense budgets. The program had mixed success in 2010, achieving 6 of 12 major goals it established and making varying degrees of progress on the others. Successes included the first flight of the carrier variant, award of a fixed-price aircraft procurement contract, and an accelerated pace in development flight tests that accomplished three times as many flights in 2010 as the previous 3 years combined. However, the program did not deliver as many aircraft to test and training sites as planned and made only a partial release of software capabilities. The short take off and landing variant (STOVL) experienced significant technical problems and did not meet flight test expectations. The Secretary of Defense directed a 2-year period to evaluate and engineer STOVL solutions. After more than 9 years in development and 4 in production, the JSF program has not fully demonstrated that the aircraft design is stable, manufacturing processes are mature, and the system is reliable. Engineering drawings are still being released to the manufacturing floor and design changes continue at higher rates than desired. More changes are expected as testing accelerates. Test and production aircraft cost more and are taking longer to deliver than expected. Manufacturers are improving operations and implemented 8 of 20 recommendations from an expert panel, but have not yet demonstrated a capacity to efficiently produce at higher production rates. Substantial improvements in factory throughput and the global supply chain are needed. Development testing is still early in demonstrating that aircraft will work as intended and meet warfighter requirements. Only about 4 percent of JSF capabilities have been completely verified by flight tests, lab results, or both. Only 3 of the extensive network of 32 ground test labs and simulation models are fully accredited to ensure the fidelity of results. Software development--essential for achieving about 80 percent of the JSF functionality--is significantly behind schedule as it enters its most challenging phase.
section_title: What GAO Recommends
paragraphs: To sustain a focus on accountability and facilitate tradeoffs within the JSF program, GAO recommends that DOD (1) maintain annual funding levels at current budgeted amounts; (2) establish criteria for evaluating the STOVL's progress and make independent reviews, allowing each variant to proceed at its own pace; and (3) conduct an independent review of the software development and lab accreditation processes. DOD concurred.

section_title: Letter
paragraphs: The F-35 Lightning II, also known as the Joint Strike Fighter (JSF) is the  Department of Defense’s (DOD) most costly and ambitious aircraft  acquisition, seeking to simultaneously develop and field three aircraft  variants for the Air Force, Navy, Marine Corps, and eight international  partners. The JSF is the core of DOD’s long term tactical aircraft  recapitalization plans as it is intended to replace hundreds of legacy  aircraft. Total U.S. planned investment in the JSF is about $385 billion to  develop and acquire 2,457 aircraft through 2035. With that many dollars at  stake amidst pressing warfighter requirements for this new 5th generation  capability, the Department has lately recognized numerous technical,  financial, and management shortcomings and continues to significantly  restructure the program, adding more time and money and making other  changes that we support.
GAO has reported on JSF issues for a number of years. Appendix 2  summarizes key findings and recommendations from that body of work.  One recurring theme has been the “single step,” or revolutionary,  acquisition strategy the JSF program adopted to develop and acquire full  combat capabilities on a very aggressive, risky schedule with substantial  concurrency, or overlap among development, testing, and production  activities. That strategy, coupled with a management environment that  was slow to acknowledge and address problems with needed changes, are  prime contributors to the relatively poor cost, schedule, and performance  outcomes experienced to date. Our March 2010 report discussed  additional cost and schedule pressures, unsatisfactory performance in  manufacturing and delivering aircraft, and concerns about not meeting  warfighter requirements on time and in quantity. We recommended that  (1) DOD complete an independent, comprehensive cost and schedule  estimate and establish it as the official program of record for planning,  budgeting, and congressional reporting purposes and (2) reassess  warfighter requirements to determine the minimum needs (both  capabilities and capacity) that can be achieved within realistic and  reasonable timeframes and, if necessary, defer some capabilities to future  increments. In addition, we suggested that Congress consider requiring  DOD to establish a “system maturity matrix,” a management tool to better  measure the program’s annual progress toward key objectives to improve  oversight and better inform future budget decisions.
This is our second report required by law in which we (1) evaluate  program cost and schedule changes and their implications on affordability;  (2) identify progress made in 2010 against established goals; (3) assess  elements of design stability and manufacturing maturity and review  production results; and (4) report status of development testing and  technical challenges facing the program.  To conduct this work, we  evaluated DOD’s restructuring actions and impacts on the program,  tracked cost and schedule changes, and determined factors driving the  changes. We reviewed program status reports, manufacturing data, test  plans, and internal DOD analyses.  We discussed results to date and future  plans to complete JSF development and move further into procurement  with DOD, JSF, and contractor officials including members of the  independent review teams. We toured aircraft and engine manufacturing  plants, obtained production and supply performance indicators, and  discussed improvements underway with contractors.  We conducted this  performance audit from May 2010 to March 2011 in accordance with  generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on our  audit objectives.
subsections: section_title: Background
paragraphs: The F-35 program is a joint, multinational acquisition to develop and field  an affordable, highly common family of stealthy, next-generation strike  fighter aircraft for the United States Air Force, Marine Corps, Navy, and  eight international partners. The JSF is a single-seat, single-engine aircraft  incorporating low-observable (stealth) technologies, defensive avionics,  advanced sensor fusion, internal and external weapons, and advanced  prognostic maintenance capability. There are three variants. The F-35A  conventional takeoff and landing (CTOL) variant will provide air-to-ground  attack capabilities to replace the Air Force’s F-16 Fighting Falcon and the  A-10 Thunderbolt II aircraft, and will complement the F-22A Raptor. The   F-35B short takeoff and vertical landing (STOVL) aircraft will be a multi- role strike fighter to replace the Marine Corps’ F/A-18C/D Hornet and AV- 8B Harrier aircraft. The F-35C carrier-suitable variant (CV) will provide the  Navy and Marine Corps a multi-role, stealthy strike aircraft to complement  the F/A-18E/F Super Hornet. The JSF is DOD’s largest cooperative  program. Our international partners are providing about $5.1 billion  toward development, and foreign firms are part of the industrial base  producing aircraft. DOD’s funding requirements for the JSF assume  economic benefits from partner purchases in reducing unit costs for U.S.  aircraft.
JSF concept development began in November 1996 with a 5-year  competition between contractors to determine the most capable and  affordable preliminary aircraft design. Lockheed Martin won the  competition and the JSF program entered system development and  demonstration in October 2001. Pratt and Whitney is the primary engine  manufacturer, while General Electric has been developing a potential  second source for the engine. System integration efforts and a preliminary  design review then revealed significant airframe weight problems  impacting key performance requirements. In March 2004, DOD rebaselined  the program, adding time and money for development and delaying key  milestones. The Navy and Marine Corps also reduced their planned  procurement by 409 jets, reducing the total U.S. buy to the current 2,457  quantity. The program was again rebaselined in March 2007 to reflect  additional cost increases and schedule slips and the procurement period  was extended by 7 years to 2034 with reduction in annual quantities.
Because of continuing problems and poor outcomes, the Secretary of  Defense announced another comprehensive restructuring of the JSF  program in February 2010.  The restructuring followed an extensive  Department-wide review initiated in 2009 and considered the findings and  recommendations from three independent groups chartered to assess the  program: the Joint Estimating Team (JET) evaluated program execution  and resource requirements; the Independent Manufacturing Review Team  (IMRT) assessed contractor capabilities and plans for ramping-up and  sustaining production at maximum rates; and the Joint Assessment Team  (JAT) reviewed engine costs and affordability initiatives.  Key  restructuring actions included adding $2.8 billion for development,  extending flight testing by 13 months, adding flight test resources (one  new test jet and use of 3 production jets), reduced near-term procurement  by 122 aircraft, and review of the military services’ capability need dates.  The Under Secretary of Defense for Acquisition, Technology and Logistics  stated that the department-wide review would continue under new  program management and cited 2010 as a critical year for assessing  progress against the new plans and the expected delivery of all test  aircraft, completion of hundreds of test flights, and meeting other key  milestones.  We supported these actions in our March 2010 report and  subsequent testimonies.  We noted the likelihood of additional cost  growth and schedule extensions as the restructuring continues.
In March 2010, the Department declared that the program experienced a  breach of the critical cost growth statutory thresholds.  The Department  subsequently certified to Congress in June 2010 that the JSF program  should continue. Table 1 summarizes the evolution of JSF cost and  schedule estimates at key junctures in its acquisition history through the  current Nunn-McCurdy certification.  Since then, in January 2011, the  Secretary of Defense announced additional development cost increases  and further changes consequent to the ongoing restructure, but has not yet  established a new approved acquisition program baseline.
section_title: Affordability Expectations Are Challenged as JSF Acquisition Costs Rise and Schedules Slip
paragraphs: Ongoing JSF restructuring continues to add more cost and time for  developing, testing, and delivering aircraft to the warfighter.  These  actions, effectively implemented, should result in more predictable and  achievable program outcomes, but restructuring comes with  consequences—higher upfront development costs, fewer aircraft received  in the near term, training delays, and extended times for testing and  delivering the capabilities required by the warfighter. Affordability for the  U.S. and our allies is challenged because unit prices are about double what  they were at program start and with new forecasts that the aircraft may  cost substantially more to operate and maintain over the life cycle than the  legacy aircraft they replace. Going forward, the program requires  unprecedented levels of funding in a period of more austere defense  funding. Defense leaders stated that the JSF program lost its focus on  affordability and that restoring the focus is paramount to improving  program outcomes.
subsections: section_title: Additional Cost Increases and Schedule Delays Are Expected as Program Restructuring Continues
paragraphs: Defense leadership continued to restructure the JSF program following  the Nunn-McCurdy certification. In January 2011, the Secretary of Defense  directed additional changes, stemming in large part from the results of a  comprehensive technical baseline review under new government and  contractor management.  Key program changes (1) added $4.6 billion to  the development program through completion for a total development  program estimate of $56.4 billion (an increase of 26 percent against the  current baseline and 64 percent from the original baseline at program  start); (2) extended the development test period to 2016 (a 4-year slip from  the current baseline); and (3) reduced near-term procurement quantities  by 124 aircraft in addition to the 122 aircraft cut announced in February  2010; and (4) lowered the annual  rate of increase for boosting future  production.
Because of the lingering technical issues on the STOVL, the most complex  variant, the Secretary decoupled STOVL flight tests from the combined  test plan and scaled back STOVL production to only 3 in fiscal year 2011  and to 6 per year for fiscal years 2012 and 2013. This represents a total cut  of 37 STOVL aircraft during this 3-year period compared to the fiscal year  2011 budget plans. In announcing these changes, the Secretary also noted  the STOVL’s significant testing problems which include lift fan engine  deficiencies, and poor durability test results, which could require  redesigns and add weight to aircraft’s structure and propulsion system.  While the Secretary decoupled STOVL from the flight test program, STOVL  was not further separated from the rest of the JSF program for  management and reporting activities.  It remains a part of the combined  JSF program for milestone decisions and cost, schedule, and performance  reporting.  Resolving STOVL problems and moving forward at an  affordable cost is essential to the Marine Corps’ future plans, which  depend upon acquiring the STOVL in quantity to directly accompany,  protect, and provide firepower to its ground expeditionary forces.
The recently submitted fiscal year 2012 Defense Budget reflects the  financial impacts from restructuring actions through 2016. Compared to  estimates in the fiscal year 2010 future years defense program for the same  5-year period, the Department increased development funding by   $7.7 billion and decreased procurement funding by $8.4 billion reflecting  plans to buy fewer aircraft.  Table 2 summarizes the revised development  and procurement funding requirements and annual quantities following  the Secretary’s reductions. Even after decreasing near-term quantities and  lowering the ramp rate, JSF procurement still rapidly increases.  Annual  funding levels more than double and quantities more than triple during this  period. These numbers do not include the additional orders expected from  the international partners.
Additional changes to cost and schedule are likely as restructuring  continues. At the time of this report, the Secretary had not yet granted new  milestone B approval nor approved a new acquisition program baseline.  Originally planned for November 2010, program officials now expect the  new acquisition program baseline in late 2011. Also, cost analysts are still  revising procurement funding requirements for the period fiscal year 2017  through completion of procurement in 2035. Accordingly, the net effect of  reducing near-term procurement quantities and deferring these aircraft to  future years is uncertain and depends upon the assumptions made about  future unit prices, annual quantities, and inflation. We expect total  procurement costs will be somewhat higher than the estimate submitted in  the Nunn-McCurdy certification (refer to table 1). Reduced quantities and  use of production aircraft in testing will also limit training activities for the  near-term and delay deliveries of new capabilities to the warfighters.   Officials now forecast that the completion of system development,  completion of initial operational testing, and the full rate production  decision will extend into 2018.  This represents slips of about 5 years in  these important milestones against the current program baseline approved  in 2007.
The military services are evaluating the impacts from restructuring on  their initial operational capability (IOC) milestones, the critical need dates  when the warfighter must have in place the first increment of operational  forces available for combat.  In response to the initial set of restructuring  actions, the Air Force and Navy tentatively extended these milestones to  2016, but the Marine Corps slightly adjusted its IOC date by 9 months to  December 2012. It is all but certain that the Marine Corps will be delaying  its IOC date in the wake of the Secretary’s STOVL actions. Air Force and  Navy dates may also be adjusted to reflect the newest developments.
section_title: Program Requires Unprecedented Funding Levels Well into the Future
paragraphs: Affordability—both in terms of the investment costs to acquire the JSF  and the continuing costs to operate and maintain it over the life-cycle—is  at risk. A key tenet of the JSF program from its inception has been to  deliver an affordable, highly common fifth generation aircraft that could  be acquired by the warfighters in large numbers. Rising aircraft prices  erode buying power and make it difficult for the U.S. and its allies to buy  as many aircraft as planned.  Quantity reductions could drive additional  price increases for future aircraft. Further, while the Department is still  refining cost projections for operating and supporting future JSF fleets,  cost forecasts have increased as the program matures and more data  becomes available. Current JSF life-cycle cost estimates are considerably  higher than the legacy aircraft it will replace; this has major implications  for future demands on military operating and support budgets and plans  for recapitalizing fighter forces. Defense leadership stated that the JSF  program lost focus on affordability and that restoring and maintaining that  focus is paramount to improving program outcomes.
In light of continued cost growth, the program places unprecedented  demands for funding in the defense budget—an annual average of almost  $11 billion for the next two decades. (This and other data in this paragraph  reflect the fiscal year 2011 budget submission.) During the peak years of  production, the average annual requirement is about $13 billion. The JSF  will have to annually compete with other defense and nondefense  priorities for the shrinking discretionary federal dollar amid continued  concerns about the national debt and long term fiscal pressures. The JSF  program has received more than $56 billion through fiscal year 2010. To  complete the acquisition program as currently planned, another             $272 billion will be required from 2011 through 2035. Figure 1 illustrates  the annual funding requirements outlined in the program’s Selected  Acquisition Report released in April 2010. These funding levels do not  reflect the additional funding increases in the Nunn-McCurdy certification  and the Secretary’s recent actions. DOD is in the process of establishing a  new acquisition program baseline which will likely project even higher  funding requirements.
The JSF is the linchpin in DOD’s tactical aircraft recapitalization plans,  replacing hundreds of legacy aircraft. Because of its sheer size and high  priority within the Department, even relatively modest cost growth on the  JSF can require the sourcing of billions of additional funds, largely from  other programs in DOD’s acquisition portfolio. On the other hand, slips in  JSF schedules, cuts in annual procurement quantities, and deferred  delivery of operational aircraft can require additional monies be spent on  legacy aircraft, postponing planned retirements and sustaining fleets for  longer periods of time.  To mitigate projected shortfalls in tactical aircraft  inventories due to JSF perturbations, the Navy recently procured  additional F/A-18E/F Super Hornets and both the Navy and Air Force are  funding service life extension programs and adding new capabilities to  legacy aircraft.
Furthermore, international partners’ participation in the JSF program is  very important to maintaining affordability for all buyers. DOD budget  plans expect the partners to buy 223 aircraft costing $24.1 billion during  the fiscal year 2011-2016 period. However, JSF cost increases, schedule  delays, and internal issues may result in reduced or deferred foreign buys.   Some partners have already signaled plans to buy fewer aircraft, a  different mix of aircraft, or defer purchases to later years. On the positive  side, other countries have expressed interest in acquiring the JSF.
Decisions made by the international community and its impact on JSF  affordability are largely beyond the program’s direct control. However,  improving JSF program outcomes to lower costs and reassure buyers is  within DOD’s and the contractors’ control.
The eight international partners have important stakes in the JSF program,  having provided about $5 billion in development funding, being counted  upon to procure hundreds of aircraft, and expecting their industries to  receive a significant portion of JSF manufacturing and supply business.   DOD’s procurement cost estimates provided to the Congress have long  assumed that the eight partners will buy at least 730 JSF aircraft.  Unit  prices for U.S. quantities assume the economic benefit of these purchases.   If fewer are sold overseas, the Air Force, Navy and Marine Corps (and the  American taxpayer) may have to pay more. Unit costs can be expected to  increase with smaller purchases due to diminished manufacturing  economies of scale and because fixed costs have to be spread over fewer  aircraft.
Maintaining a strong focus on affordability necessitates having reliable and  complete cost data that provides accurate accounting reports, identifies  potential cost and schedule problems early, and produces sound estimates  of the cost to complete work. The JSF program has been hampered in this  regard because, for at least the past three years, the prime contractor has  not had an adequate and disciplined earned value management (EVM)  system in place to effectively track costs and control schedule. The prime  contractor was found deficient in meeting 19 of 32 required guidelines,  calling into question its ability to manage the escalating costs and complex  scheduling of the JSF program.  In October 2010, the Defense Contract  Management Agency (DCMA) withdrew the determination of compliance  for the prime contractor’s EVM system due to longstanding non- compliance issues with specific guidelines that underpin a sound system.   To address these shortcomings, the contractor is developing new  processes, tools, training, and enforcement in order to achieve a fully  integrated and automated EVM system. Officials will reassess the earned  value system by March 2012—more than four years after these problems  were first discovered to see if modifications needed have been made.
EVM is an important, established tool that can provide objective product  status reports. DOD requires its use by major defense suppliers to  facilitate good insight and oversight of the expenditure of government  dollars, thereby improving both affordability and accountability.  JSF is  DOD’s largest acquisition ever, so it is particularly critical to improve and  certify the contractor’s EVM system as expeditiously as possible.  If not  improved, inaccurate performance reports and late notice of cost overruns  will likely continue to hinder timely decision making and corrective  actions.  Strong leadership and a shared vision among stakeholders are  critical to implementing EVM effectively.
section_title: Progress in Achieving the JSF Program’s 2010 Goals Was Mixed
paragraphs: The JSF program established 12 clearly-stated goals in testing, contracting,  and manufacturing for completion in calendar year 2010. It had mixed  success, achieving 6 goals and making varying degrees of progress on the  other 6.  For example, the program exceeded its goal for the number of  development flight tests but did not deliver as many test and production  aircraft as planned.  Also, the program awarded its first fixed-price  contract on its fourth lot of aircraft production, but did not award the  fixed-price engine contract in 2010 as planned. Table 3 summarizes JSF  goals and accomplishments for 2010.
The development flight test program significantly ramped up operations in  2010, accomplishing three times as many test flights as the previous   3 years combined.  Table 4 summarizes actual flights, hours, and test  points flown by each variant compared to the 2010 plan.
Although still hampered as in prior years by the late delivery of test  aircraft, flight tests substantially increased in volume and pace at the two  main government test sites—Edwards Air Force Base, California, for  CTOL tests and Patuxent River Naval Air Station for STOVL and CV  testing. The CTOL variant significantly exceeded plans while initial testing  of the carrier variant was judged satisfactory, below plans for the number  and hours of flight but ahead on test points flown.  The STOVL, however,  substantially under-performed in flight tests and experienced significant  technical issues unique to this variant that could add to its weight and  cost.  The STOVL’s test problems were a major factor in the heightened  scrutiny and two-year probation period directed by the Secretary to  engineer solutions, assess impacts, and inform a future decision as to  whether and how to proceed with this variant.
Evaluating annual performance against stated goals can be an effective  tool that facilitates oversight by the Congress and defense leadership and  useful for informing future budget decisions. In our 2010 report, we  suggested that Congress consider requiring DOD to establish a system  maturity matrix to better measure the program’s progress in maturing the  weapon system and providing evidence to support budget decisions.  The  Ike Skelton National Defense Authorization Act for Fiscal Year 2011  established this requirement and we understand the Department is  working on its implementation. We believe this tool and process will  improve oversight and budgeting, holding people accountable for meeting  interim objectives and, for objectives not met, providing criteria and a  forum for evaluating reasons why and what should be done.
section_title: Program Has Still Not Fully Demonstrated a Stable Design and Mature Manufacturing Processes as It Enters Its Fifth Year of Production
paragraphs: After completing 9 years of system development and 4 years of  overlapping production activities, the JSF program has been slow to gain  adequate knowledge that its design and manufacturing process are fully  mature and ready for greater levels of annual production. The JSF  program still lags in achieving critical indicators of success expected from  well-performing acquisition programs.  Specifically, the program has not  yet stabilized aircraft designs—engineering changes continue at higher  than expected rates long after critical design reviews and well into  procurement, and more changes are expected as testing accelerates. Also,  the aircraft and engine manufacturing processes are not yet mature  enough to support efficient production at higher annual rates and  substantial improvements in the global supply network are needed.   Further, the growth in aircraft reliability—crucial for managing life-cycle  costs—has not been demonstrated to the extent planned by this time.
subsections: section_title: The Program Has Not Yet Stabilized Aircraft Designs
paragraphs: Engineering drawings released since design reviews and the number and  rate of design changes are excessive compared to plans and best practices.   Critical design reviews were completed on the three aircraft variants in  2006 and 2007 and the designs declared mature, but the program continues  to experience numerous changes.  Since 2007, the program has produced  20,000 more engineering drawings, a 50-percent increase in total drawings  and about 5 times more than best practices suggest.  In addition, changes  to drawings have not decreased and leveled off as planned.  Figure 2  tracks and compares monthly design changes and future forecasts against  contractor plans in 2007.  The monthly rate in 2009 and 2010 was higher  than expected and the program now anticipates more changes over a  longer period of time—about 10,000 more changes through January 2016.   We expect this number to go up given new forecasts for additional testing  and extension of system development until 2018.
A key indicator of a product’s maturity is the stability of its design.  The  number of engineering drawings released and subsequent changes provide  indicators of the maturity of the design.  Engineering drawings are critical  because they communicate to the manufacturer and suppliers how the  part functions, what it looks like, and what materials and critical  processes are used to build the product.  Best practices suggest 90 percent  of a product’s engineering drawings be released by the critical design  review. Late engineering drawings and high levels of changes often  indicate a lack of understanding about the design, and can cause part  shortages and inefficient manufacturing processes as work is performed  out of sequence. Some level of design change is expected during the  production cycle of any new and highly technical product, but excessive  changes raise questions about the JSF’s design maturity and its readiness  for higher rates of production.
With most of development testing still ahead for the JSF, the risk and  impact from required design changes are significant.  Acquisition programs  typically encounter higher and more substantive changes as a result of  discovery and rework during development flight and ground testing.   Future changes may require alterations to the manufacturing process,  changes to the supply base, and costly retrofitting of aircraft already  produced and fielded. A key cost driver for the program has been the  higher than expected effort needed to address design related issues. The  contractor has not been able to reduce engineering staff as fast as  expected.  DOD’s restructuring actions recognize these issues and added  time to development, more flight testing, and reduced procurement.  Additional changes are likely as development flight testing continues.
Some emerging concerns may drive additional and substantive design  changes:       JSF Lift System Development and Integration.  Essential to  STOVL operations, the lift fan continues to be a prime risk area. The  program is working to mature lift fan and drive shaft technologies and  a required redesign expected in spring 2011.
Fatigue Cracks in STOVL Test Article. During a recent durability  ground test, fatigue cracks were discovered in a major bulkhead of the  STOVL test article. Cracks were discovered after 1,500 hours of  durability testing, less than one-tenth of the hours planned for fatigue  tests to certify that the STOVL airframe meets its design life  requirement. Officials reported that stress data had been under- estimated during initial design. Inspections of aircraft and other test  articles did not identify cracks at the same site. Decisions about  potential redesign and re-manufacture are still to be determined.
Wing Tip Vortex. Prime contractor officials identified wing tip  vortices as a potential risk to the program. Wing tip vortices are tubes  of circulating air which are left behind the aircraft’s wing as it  generates lift. The cores of the vortices are sometimes visible because  of water condensation. If these are visible during daytime flights they  could negatively impact the aircraft’s stealth capabilities.
Outer Mold Lines. Defense Contract Management Agency officials  noted difficulties in manufacturing outer mold lines, resulting from  tight tolerance specifications and multiple manufacturing  methodologies among the different JSF parts suppliers. The  manufacturing processes are new and different from legacy practices.   Inability to meet the outer mold line requirements could have major  impacts on cost as well as stealth requirements and capabilities.  This  problem is not expected to be resolved until the June 2015 time frame  after which a large number of aircraft will have been built and would  need to be retrofitted for any design changes. Program officials stated  some redesign activities have begun and will take into the 2013  timeframe to begin developing the changes, their costs, and  implementation. The effects of these changes could extend out into  2015, but will be prioritized to reduce performance and cost impacts.
section_title: Manufacturing Processes are Not Yet Mature Enough for Efficient Production at Increased Rates
paragraphs: Manufacturing and delivering test jets took much more time and money  than planned and the full contingent of test aircraft is still not available at  military testing sites, years later than promised. Projected costs to  complete the first three production lots for aircraft and engines also  exceed the negotiated amounts at contract award and aircraft will be  delivered late. The production impacts of restructuring actions that  reduced quantities, lowered the ramp rate, and delayed the full-rate  production decision have not been fully determined. We found that the  aircraft and engine manufacturers are making good faith efforts to  implement the IMRT and JAT recommendations and to make other  improvements with performance measures indicating some success.
As in prior years, lingering management inefficiencies, including  substantial out-of-station work and part shortages, continued to increase  the labor needed to manufacture test aircraft. Figure 3 depicts forecasted  and actual labor hour requirements for building 12 production- representative test jets. Total labor hours required to produce the test  aircraft increased over time.   The 2010 actual labor hours exceeded the  2007 budgeted hours by more than 1.5 million hours, a 75 percent increase.
Manufacturing production aircraft is different from building test aircraft,  and some gains in learning as more aircraft are built can be expected to,  over time, reduce labor hour costs. However, the experience to date on the  test aircraft and initial production aircraft suggests that future costs for  building production aircraft may be higher than currently budgeted. The  costs on the first three low-rate production contracts have increased from  amounts negotiated at contract award and the completion dates for  delivering aircraft have been extended over nine months on average. We  are encouraged by DOD’s award of a fixed price incentive fee contract for  lot 4 production and the prospects for the cost study to inform lot 5  negotiations, but we have not examined contract specifications.
DOD began procuring production jets in 2007 and has now ordered 58  aircraft on the first four low-rate initial production lots. JSF contracts  anticipated the delivery of 14 production jets through 2010, but none have  been delivered. Delivery of the first two production jets (both CTOLs) has  been delayed several times since the contract was signed and is now  expected in April 2011. In addition, DOD expects to procure 32 more  aircraft in fiscal year 2011. Building a large backlog of jets on order but  undelivered is not an efficient use of federal funds, tying up millions of  dollars in obligations ahead of the ability of the manufacturing process to  produce.  We note that the Secretary used a similar line of reasoning to  reduce STOVL production.
DOD does not yet know the full effect that restructuring actions will have  on future annual procurement funding requirements.  Cost analysts are  still calculating the impacts from deferring procurement of 246 aircraft  from the near-term to future years, lowering the ramp rate, and extending  the full-rate production decision.  Future funding requirements could be  even higher than projected and the quantities considered affordable by the  U.S. and allies could be reduced, further driving up unit costs.
The Secretary’s decisions to reduce near-term procurement quantities and  adopt a less-steep ramp up in future production were based on IMRT  findings. The Secretary chartered the IMRT to comprehensively review  JSF manufacturing capacity to assess the contractor’s ability to achieve  planned production ramp-up and to sustain the predicted maximum  production rates.  The IMRT’s October 2009 report made 20 specific  recommendations for corrective actions.  As of September 2010, officials  considered eight of the recommendations complete and three others on  track. Implementation of the remaining nine recommendations was  incomplete or behind schedule.
The most significant incomplete recommendation is improving global  supply chain management.  The JSF already has an extensive number of  suppliers worldwide and those numbers will increase with future  workload shared among numerous domestic and foreign firms. The IMRT  cites the global supply chain as the critical manufacturing challenge facing  the program, requiring significant improvement in delivery performance  and responsiveness in order to achieve the program’s eventual production  rate goal of 20 aircraft per month.  According to the prime contractor, the  global supply chain remains on the critical path and progress has been  made, but the global transportation plan and supply chain risk  management plan are incomplete. Another IMRT recommendation that  still needs to be addressed is the performance of a comprehensive  schedule risk assessment, now expected to begin in spring 2011. We  recommended this in our March 2009 report.  Schedule risk assessments  can provide keen insight into critical path activities, cost and schedule  interrelationships, and emerging risks.
The primary F135 engine contractor faces similar challenges as it moves  deeper into production. All development engines and initial production  units have been delivered, but the costs to complete each of the first three  engine production contracts increased and deliveries slipped since  contract awards.  Officials said these delays have not been especially  troublesome to date because aircraft deliveries were even later.  The  contractor achieved the initial service release for the CTOL and CV engine,  meaning the engine configuration is qualified and ready to go into  production, but the STOVL’s initial release was delayed until December  2010 due to qualification testing.  The JAT reviewed F135 program  performance, identified cost drivers, and made affordability projections.   JAT officials said the contractor’s cost reduction efforts were credible but  largely dependent on receiving more government funding for affordability  initiatives and alternative sourcing arrangements.
Our past work in best practices found that successful product  development programs reach a point at which they know that  manufacturing processes will efficiently produce a new product  conforming to cost, quality, and schedule targets before they begin  producing a system.  Reaching this point means more than knowing that  the product can be built; it means that critical manufacturing processes  are under control, such that the quality, volume, and cost are proven  acceptable. By these criteria, the JSF contractors’ abilities to ramp-up to  greater rates of production have not yet been demonstrated. The aircraft  and engine manufacturers now have significantly more items in  production flow compared to prior years, but throughput capacity to  complete all work and deliver end items is constrained.  We determined  that the aircraft and engine contractors are making good faith efforts to  implement the recommendations of the IMRT and JAT and to make other  improvements to production capacity and flow.  The aircraft manufacturer  is reporting a decrease in out of station work, more efficient work stations,  improved quality, increased parts availability, and reduced span times.  Until improvements are fully implemented and demonstrated, the  restructuring actions to reduce near term procurement quantities and  establish a more achievable ramp rate was appropriate and will provide  more time to fully mature manufacturing and supply processes and catch  up with aircraft backlogs.  Improving factory throughput and controlling  costs—driving down unit costs and delivering on time— are essential for  efficient manufacturing and timely delivery to the warfighter at the  increased production rates planned for the future.
section_title: Aircraft Are Not Meeting Early Reliability Growth Plans
paragraphs: STOVL and CTOL aircraft are behind reliability growth plans aimed at  demonstrating that the aircraft will meet warfighter support and  availability requirements. The carrier variant is in early stages of flight  testing and sufficient reliability data was not available.  Reliability is a  function of the specific elements of a product’s design; a system is reliable  when it can perform over a specified period of time without failure,  degradation, or need of repair. Improvements over time occur through  design changes or manufacturing process improvements. A key reliability  metric is mean flying hours between failure, defined as the number of  flying hours achieved divided by the number of failures incurred.   Reliability growth plans called for the STOVL to have achieved at least     1.9 flying hours between failures and for the CTOL 2.9 flying hours  between failures by this point in the test program.   However, the STOVL  aircraft is significantly behind plans, achieving about 0.4 hours between  failures, or about 20 percent of what was expected by this time.  The CTOL  variant was also behind plans achieving 1.8 hours between failures,  approximately 60 percent of what was expected.  Figure 4 depicts progress  of each variant in demonstrating mean flying hours between failures, as of  September 2010.
Improving reliability rates are essential to control future operating costs  and ensure aircraft are available as needed by the warfighter. Compared to  the up-front costs of acquiring aircraft, the long-term costs for operating,  maintaining, and sustaining JSF fleets over an aircraft’s useful life  represent the much larger portion of total ownership costs.  We have  reported in the past that it is important to demonstrate that system  reliability is on track to meet goals before production begins as changes  after production commences can be inefficient and costly.
section_title: Testing Has Been Slow and Has Not Demonstrated That the Aircraft Will Work in Its Intended Environment
paragraphs: The JSF program is still very early in demonstrating aircraft design and  testing to verify it works as intended.  As of December 2010, about four  percent of JSF capabilities have been completely verified by flight tests,  lab results, or both.  Initial tests of a fully integrated aircraft to  demonstrate full mission systems capabilities and weapons delivery is now  not expected until 2015, three years later than planned. The program  demonstrated measurable progress in development flight testing during  2010, but still lags earlier expectations, and the STOVL problems have  constrained overall progress. Only 3 of 32 ground test labs and simulation  models critical to complement and, in some cases, substitute for flight  tests, are accredited to verify and ensure the fidelity of results. Software  development—essential for achieving about 80 percent of the JSF  functionality—is significantly behind schedule as it enters its most  challenging phase.  Software delivery to the test program that is essential  to demonstrating full system capability is now expected in late 2014, a   3-year delay.
Our work in best practices suggests that a key indicator of a product’s  maturity and readiness for production is when a fully integrated, capable  system has been demonstrated to work in its intended environment.  A  fully integrated, capable system would include the integration of all the  hardware, including mission avionics systems, and software needed to  provide the system its full mission capabilities.  Many past DOD weapons  programs have failed to demonstrate that the system works as intended  before entering production, discovering costly design problems late in  development when the more complex software and advanced capabilities  are integrated and tested.
subsections: section_title: Development Flight Testing Is Progressing but Behind Plans
paragraphs: Development flight testing was much more active in 2010 than prior years  and had some notable successes, but overall still lagged behind  expectations. The continuing effects from late delivery of test aircraft and  an inability to achieve the planned flying rates per aircraft substantially  reduced the amount and pace of testing planned previously. Consequently,  even though the flight test program accelerated its pace last year, the total  number of flights accomplished during the first four years of the test  program significantly lagged expectations when the program’s 2007  baseline was established.  Figure 5 shows that the cumulative number of  flights accomplished by the end of 2010 was only about one-fifth the  number forecast by this time in the 2007 test plan.
Program officials reported that 13 test aircraft are now out of production.   Ten test aircraft have been ferried to test sites and others are in varying  stages of final check-out. The program has accomplished first flights for all  three variants. Officials had hoped aircraft could achieve a rate of 12  flights per month.  However, the average flight rate for 2010 ranged from  over 2 to almost 8 per month.
By the end of 2010, about 10 percent of more than fifty thousand planned  test points have been completed. According to program officials,  completion of a test point means that the test point has been flown and  that flight engineers ruled that the point has met the need. Further analysis  may be necessary for the test point to be closed out. The majority of the  points were earned on airworthiness tests (basic airframe handling  characteristics) and in ferrying the planes to test sites.  According to a  senior level DOD test official, airworthiness and ferry test points should be  relatively easy to accomplish. Remaining test points include more complex  and stringent requirements, such as mission systems, ship suitability, and  weapons integration that have yet to be demonstrated.
As discussed earlier, STOVL flight performance lagged plans during 2010,  while the CTOL variant exceeded and the CV variant generally met plans.  Officials reported that design and manufacturing defects and excessive  component failures caused prolonged maintenance periods that drove the  low fly rates. For instance, in the July to August 2010 period, STOVL test  aircraft were down for unscheduled maintenance more than half the time.  Further test delays will likely cause the program to miss critical future  milestones. STOVL initial at-sea testing will not start until October 2011  because of delays in clearing the vertical-landing envelope. STOVL-related  delays are also causing Marine Corps leadership to reassess its  requirements and will likely extend the date for achieving initial  operational capabilities, currently set in December 2012.
Concerned that STOVL testing problems were negatively affecting the  other variants, the Department moved to decouple the STOVL testing and  placed the variant on a two-year probation period to work out problems  and get back on track. The Secretary’s actions will require a new test plan  since current flight test plans rely substantially on the STOVL to fly and  demonstrate test points in common with other variants. The current plan  has the STOVL responsible for completing about 43 percent of the total  test points.
JSF restructuring actions are positive and support a more robust and  achievable test plan.  Officials added more resources for development  testing, extended the flight test schedule, and reduced the overlap with  initial operational testing. More recently, officials revised the test plan  increasing the total number of test flights from 5,856 to 7,727, about one- third more. To increase capacity, the restructure added one carrier variant  test aircraft, an additional software integration line, and allowed the  program to utilize up to three production aircraft for development testing.   Compared to the previous test plan, officials assumed more ground time  for aircraft maintenance and planned modifications, as well as a more  measured ramp-up in the rate of flights per test aircraft.  The restructuring  largely reverses the program’s earlier Mid-Course Risk Reduction plan that  reduced test resources. Our March 2008 report criticized DOD’s mid- course plan, particularly the cuts made in flight test assets and the number  of flight tests, as well as the program’s failure to address root causes of  cost growth, the very reasons why officials felt the mid-course plan was  needed. Since that report was issued, JSF cost and schedule continued to  deteriorate and officials recognized a need to increase test assets and add  more flight testing.
section_title: Most Ground Test Labs and Simulation Models Are Not Accredited or Verified
paragraphs: The JSF test program relies much more heavily than previous weapon  systems on its modeling and simulation labs to test and verify aircraft  design and subsystem performance.  However, only 3 of 32 labs and  models have been fully accredited to date; the program had planned to  accredit 11 labs and models by now. Accreditation is essential to ensure  the fidelity of results validate that the models accurately reflect aircraft  performance. Accreditation is a lengthy and involved technical evaluation  using flight test data to verify lab results. Much work remains before the  program can fully utilize the models and simulation capabilities needed to  verify results and to demonstrate that ground testing can substitute for  flight testing. However, the ability to substitute is unproven and progress  in reducing program risk is difficult to assess.  Contracting officials told us  that early results are providing good correlation between ground and flight  tests.
The Director of Operational Test and Evaluation reported that 50 percent  of the models will be accredited during the final year of flight testing, a  highly risky approach. Delays in accreditation add risks to not completing  future software blocks on time and for discovering defects late. More flight  testing may be needed to cover lab shortcomings, but is generally more  expensive, and could lead to more delays in completing development and  operational testing.  It could also require more production aircraft for a  longer period to supplement test assets, resulting in fewer systems at  training sites and operational bases.
Contractor utilization of labs has increased markedly and the number and  integration of labs is impressive, but capacity may be constrained. Because  of development concurrency, there is overlap in scheduling the new  blocks and resources must be shared between blocks when rework on an  earlier block is required. If integration and test is delayed due to capacity  or conflict with an earlier block, lab officials said that expected  capabilities may not be delivered on time to meet flight test and training  dates. Mitigating strategies include adding people, lab capacity, software  test lines, and shifting capabilities to later blocks. The 2010 restructuring  added $250 million to increase integration lab capacity.  According to  program officials, the greater number of labs allows engineers to work  simultaneously on different development blocks, reducing bottlenecks  that may occur in testing. Program and contractor officials believe that the  up-front investment of $5 billion in simulation labs will pay off in early risk  reduction, reduce flights, control costs, and are essential to meet key  milestones in JSF’s aggressive test plan.
section_title: Software Development Is behind Schedule with Most Complex Work Still Ahead
paragraphs: Software providing essential JSF capability is not mature and releases to  the test program are behind schedule. Officials underestimated the time  and effort needed to develop and integrate the software, substantially  contributing to the program’s overall cost and schedule problems and  testing delays, while requiring the retention of engineers for longer  periods. Significant learning and development work remains before the  program can demonstrate the mature software capabilities needed to meet  warfighter requirements. Good progress has been made in the writing of  software code—about three-fourths of the software has been written and  integrated, but testing is behind schedule and the most complex work is  still ahead.  Program restructuring added a second software integration  line which should improve throughput.
The JSF software development effort is one of the largest and most  complex in DOD history, providing 80 percent of JSF’s functionality  essential to capabilities such as sensor fusion, weapons and fire control,  maintenance diagnostics, and propulsion.  JSF has about 8 times more on- board software lines of code than the F/A-18E/F Super Hornet and 4 times  more than the F-22A Raptor.  Also, the amount of code needed will likely  increase as integration and testing efforts intensify.  In 2009, officials  reported that about 40 percent of the software had completed integration  and testing.  They did not provide us a progress report through 2010.  Integration and test is a lengthy effort and is typically the most challenging  phase of software development requiring specialized skills and integration  test lines.  The program has experienced a growth of 40 percent in total  software lines of code since preliminary design review and 13 percent  growth since the critical design review. Other recent defense acquisitions  have experienced 30 to 100 percent growth in software over time.
Software capabilities are developed, tested, and delivered in 5 blocks, or  increments.  Several blocks have grown in size and taken longer to  complete than planned.  Software defects, low productivity, and  concurrent development of successive blocks created inefficiencies,  taking longer to fix defects and delaying the demonstration of critical  capabilities.   In addition, program and prime contractor officials  acknowledge they do not include integration as a key tracking metric and  have been unable to agree on how to track it. This has made it hard for the  program to analyze integration trends and take action to remedy the  situation. Instead the program office and prime contractor have made  several adjustments to the software development schedule, each time  lengthening the time needed to complete work, as shown in figure 7.
Delays in developing, integrating, and releasing software to the test  program have cascading effects hampering flight tests, training, and lab  accreditation. While progress is being made, a substantial amount of  software work remains before the program can demonstrate full  warfighting capability.  The program released block 0.5 for flight test  nearly 2 years later than planned in the 2006 plan, largely due to  integration problems. Each of the remaining three blocks—providing full  mission systems and warfighting capabilities—are now projected to slip  between 2 to 3 years compared to the 2006 plan. Defects and workload  bottlenecks delayed the release of full block 1 capabilities; the initial  limited release of block 1 software was flown for the first time in  November 2010. Software defects increased throughout 2010, but fixing  defects did not keep pace. Some capabilities were moved to future blocks  in attempts to meet schedule and mitigate risks.  For example, full data  fusion mission systems were deferred from block 2 to 3. Further trades  and deferrals may be needed. Rather than working all blocks concurrently,  focusing efforts on a more measured evolutionary approach could result in  more timely release of incremental capabilities to the testing, training, and  warfighter communities. Development and integration of the most  advanced capabilities could be deferred to future increments and  delivered to the warfighter at a later date.
The recent technical baseline review identified software as a significant  challenge, slowing system development and requiring more time and  money.  Although officials are confident that such risks can be addressed,  the scale and complexity of what is involved remains a technically  challenging and lengthy effort.  Uncertainties pertaining to critical  technologies, including the helmet-mounted display and advanced data  links, add to challenges. Deficiencies in the helmet mounted display,  especially latency in transmitting sensor data, are causing officials to  develop a second helmet while trying to fix the first model. Resolution  could result in a major redesign or changes in the JSF’s concept of  operations by placing limitations on the operational environment,  according to program officials.
section_title: Conclusions
paragraphs: The JSF program is at a critical juncture—9 years in development and         4 years in limited production, but still early in testing and verifying aircraft  performance. If effectively implemented and sustained, the Department’s  restructuring should place the JSF program on firmer footing and lead to  more achievable and predictable outcomes. However, restructuring comes  with a price tag—higher up-front development costs, fewer aircraft  received in the near term, training delays, and prolonged times for testing  and delivering the capabilities required by the warfighter. Reducing near- term procurement quantities lessens concurrency, but the overlap among  development, testing, and production activities is still substantial and  risky.  Development and testing activities will now overlap 11 years of  production based on the latest extension in key milestones. Flight testing  and production activity are increasing and contractors are improving  supply and manufacturing processes, but deliveries are still lagging. The  challenge in front of the aircraft and engine contractors is improving the  global supply chain and accelerating manufacturing throughput to produce  quality products in economic quantities and on time.  Slowed deliveries  have built a growing backlog of jets on order but not delivered; this is not  a good use of federal funds, tying up millions of obligated dollars much  ahead of the ability of the manufacturing process to produce. The  Secretary of Defense used similar reasoning in significantly reducing  STOVL procurement until technical issues are resolved and the  manufacturing process able to deliver efficiently and on time.
The JSF acquisition demands an unprecedented share of the Department’s  future investment funding. The program’s size and priority is such that its  cost overruns and extended schedules are either borne by funding cuts to  other programs or else drive increases in the top line of defense spending,  the latter not an attractive option in a period of more austere budgets. Up  until now, JSF problems have been addressed either with more time and  money or by deferring aircraft procurement to be borne by future years’  budgets.  It is past time to place some boundaries on the program such  that future difficulties can be managed within a finite budget by facilitating  trades within the JSF program and thereby minimizing impacts on other  defense programs and priorities.  Also, Department actions to limit STOVL  procurement, decouple it from development testing, and concentrate  efforts to resolve deficiencies are appropriate.  Given its criticality to the  Marine Corp’s future tactical aviation plans, additional steps may be  needed to set the framework and criteria for the “probation period” and to  sustain management focus on STOVL in order to better ascertain its  progress and inform future decisions.  Focused individual attention on  STOVL apart from the other two variants could allow each variant to  proceed through development and testing at its own pace.  Furthermore,  development testing is hampered both by the late delivery of software  increments and the lagging schedule for accrediting ground labs and  simulation models.  A comprehensive independent review of the software  development process and lab accreditation issues could enhance  management insight and identify opportunities for improvement in these  critical areas. We note that the previous independent teams established by  the Department significantly improved the manufacturing, engine, and cost  estimating processes.
We agree with defense leadership that a renewed and sustained focus on  affordability by contractors and the Government is critical to moving this  important program forward and enabling our military services and our  allies to acquire and sustain JSF forces in needed quantities.  Maintaining  senior leadership’s increased focus on program results, holding  government and contractors accountable for improving performance, and  bringing a more assertive, aggressive management approach for the JSF to  “live within its means” could help effectively manage growth in the  program and limit the consequences on other programs in the portfolio.  Controlling JSF future cost growth would minimize funding disruption and  help stabilize the defense acquisition portfolio by providing more certainty  to financial projections and by facilitating the allocation of remaining  budget authority to other defense modernization programs.
section_title: Recommendations for Executive Action
paragraphs: Given the other priorities that DOD must address in a finite budget, a  renewed and sustained focus on affordability by contractors and the  Government is critical for successfully moving the JSF program forward.   DOD must plan ahead for a way to address and manage JSF challenges  and risks in the future.  To facilitate making tradeoff decisions with  respect to the JSF program that limit impacts to other DOD programs and  priorities and to improve key management processes, we recommend that  the Secretary of Defense take the following actions to reinforce and  strengthen program cost controls and oversight:   1.  The JSF program should maintain total annual funding levels for  development and procurement at the current budgeted amounts in the  fiscal year 2012-2016 future years defense plan (modified, if warranted,  by the new acquisition program baseline expected this year). It should  facilitate trades among cost, schedule, requirements, and quantities to  control cost growth.  Having gone through the Technical Baseline  Review (TBR) and budget approval process, it is reasonable to expect  the program to execute against the future years defense plan going  forward.  Only in instances of major and unforeseen circumstances,  should the Department consider spending more money on the  program.  Even then, we would expect changes to be few and adopted  only after close scrutiny by defense leadership.  Approved changes  should be well supported, adequately documented, and reported to the  congressional defense committees.    2.  Establish criteria for the STOVL probation period and take additional  steps to sustain individual attention on STOVL-specific issues,  including independent F-35B/STOVL Progress Reviews with Senior  Leadership to ensure cost and schedule milestones are achieved to  deliver required warfighter capabilities. The intent is to allow each JSF  variant to proceed and demonstrate success at its own pace and could  result in separate full-rate production decisions.  3.  The Department should conduct an independent review of the  contractor’s software development, integration, and test processes— similar to its review of manufacturing operations—and look for  opportunities to streamline software efforts. This review should  include an evaluation of the ground lab and simulation model  accreditation process to ensure it is properly structured and robustly  resourced to support software test and verification requirements.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided us with written comments on a draft of this report. The  comments are reprinted in appendix III.  We worked collaboratively with  defense officials to hone our draft recommendations, making them more  targeted. DOD concurred with the recommendations as amended. We also  incorporated technical comments as appropriate.
We are sending copies of this report to the Secretary of Defense; the  Secretaries of the Air Force and Navy; the Commandant of the Marine  Corps; and the Director of the Office of Management and Budget. The  report also is available at no charge on the GAO Web site at  http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this report. Staff members making key contributions to this  report are listed in appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine the Joint Strike Fighter (JSF) program’s progress in meeting  cost, schedule, and performance goals, we received briefings by program  and contractor officials and reviewed financial management reports,  budget documents, annual Selected Acquisition Reports, monthly status  reports, performance indicators, and other data.  We identified changes in  cost and schedule, and obtained officials’ reasons for these changes.  We  interviewed officials from the JSF program, contractors, and the  Department of Defense (DOD) to obtain their views on progress, ongoing  concerns and actions taken to address them, and future plans to complete  JSF development and accelerate procurement. At the time of our review,  the most recent Selected Acquisition Report available was dated  December 31, 2009 and released in April 2010. At the time of our review,  DOD was preparing a new acquisition program baseline for the program  which would reflect updated cost and schedule projections.
In assessing program cost estimates, we compared the official program  cost estimate in the 2009 Selected Acquisition Report and subsequent cost  estimate developed after the Nunn-McCurdy breach to estimates  developed by the JSF program and Defense Contract Management Agency  (DCMA) reports. We interviewed program office officials and members of  the DOD Cost Analysis and Program Evaluation Office (CAPE), and DCMA  to understand their methodology, data, and approach in developing cost  estimates.  To assess the validity and reliability of contractors’ cost  estimates, we reviewed audit reports prepared by DCMA and cost  performance reports prepared by the contractor.
To access the program’s plans and risk in manufacturing and its capacity  to accelerate production, we analyzed manufacturing cost and work  performance data to assess progress against plans.  We compared  budgeted program labor hours to actual labor hours and identified growth  trends.  We reviewed data and briefings provided by the program, DCMA,  and CAPE to assess supplier performance and ability to support  accelerated production in the near term.  We also determined reasons for  manufacturing delays, discussed program and contractor plans to  improve, and projected the impact on development and operational tests.   We interviewed Naval Air Systems Command and contractor officials to  discuss Earned Value Management System issues but we did not conduct  any analysis since the data was deemed unreliable by DCMA.
To assess plans, progress, and risks in test activities, we examined  program documents and interviewed DOD, program office, and contractor  officials about current test plans and progress.  To assess progress  towards test plans, we compared the number of flight tests conducted as  of December 2010 to the original test plan established in 2007.  We also  reviewed documents and interviewed prime contractors about flight  testing, the integrated airborne test bed, and ground testing.  To assess the  ground labs and test bed, we interviewed officials and toured the testing  labs at the Lockheed Martin facilities in Fort Worth, Texas.  We also  reviewed the independent assessments conducted by the JET and NAVAIR  to obtain their perspective on the program’s progress in test activities.
In performing our work, we obtained information and interviewed officials  from the JSF Joint Program office, Arlington, Virginia; Naval Air Systems  Command, Patuxent River, Maryland; Defense Contract Management  Agency, Fort Worth, Texas; Lockheed Martin Aeronautics, Fort Worth,  Texas; Defense Contract Management Agency, Middletown, Connecticut;  and Pratt & Whitney, Middletown, Connecticut.  We also met with and  obtained data from the following offices of the Secretary of Defense in  Washington, D.C.: Director, Operational Test and Evaluation; Cost  Analysis and Program Evaluation Office; and Systems and Software  Engineering.  We assessed the reliability of DOD and JSF contractor data  by (1) performing electronic testing of required data elements,                    (2) reviewing existing information about the data, and (3) interviewing  agency officials knowledgeable about the data.  We determined that the  data were sufficiently reliable for the purposes of this report.  We  conducted this performance audit from May 2010 to February 2011 in  accordance with generally accepted government auditing standards.   Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives.  We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Prior GAO Reports on JSF and DOD Responses and Subsequent Actions
subsections: section_title: Start of system development and demonstration approved.
paragraphs: Critical technologies needed for  key aircraft performance  elements are not mature.  Program should delay start of  system development until  critical technologies are mature  to acceptable levels.
DOD did not delay start of  system development and  demonstration stating  technologies were at  acceptable maturity levels and  will manage risks in  development.
The program undergoes re- plan to address higher than  expected design weight, which  added $7 billion and 18 months  to development schedule.
We recommend that the  program reduce risks and  establish executable business  case that is knowledge-based  with an evolutionary acquisition  strategy.
DOD partially concurred but  does not adjust strategy,  believing that their approach is  balanced between cost,  schedule and technical risk.
Program sets in motion plan to  enter production in 2007 shortly  after first flight of the non- production representative  aircraft.
The program plans to enter  production with less than 1  percent of testing complete.  We recommend program delay  investing in production until  flight testing shows that JSF  performs as expected.
DOD partially concurred but did  not delay start of production  because they believe the risk  level was appropriate.
Congress reduced funding for  first two low-rate production  buys thereby slowing the ramp  up of production.
Progress is being made but  concerns remain about undue  overlap in testing and  production. We recommend  limits to annual production  quantities to 24 a year until  flying quantities are  demonstrated.
DOD non-concurred and felt  that the program had an  acceptable level of concurrency  and an appropriate acquisition  strategy.
DOD implemented a Mid- Course Risk Reduction Plan to  replenish management  reserves from about $400  million to about $1 billion by  reducing test resources.
We believe new plan actually  increases risks and that DOD  should revise the plan to  address concerns about  testing, use of management  reserves, and manufacturing.  We determine that the cost  estimate is not reliable and that  a new cost estimate and  schedule risk assessment is  needed.
DOD did not revise risk plan  nor restore testing resources,  stating that they will monitor the  new plan and adjust it if  necessary. Consistent with a  report recommendation, a new  cost estimate was eventually  prepared, but DOD refused to  do a risk and uncertainty  analysis that we felt was  important to provide a range  estimate of potential outcomes.
The program increased the  cost estimate and adds a year  to development but accelerated  the production ramp up.  Independent DOD cost  estimate (JET I) projects even  higher costs and further delays.
Because of development  problems, we stated that  moving forward with an  accelerated procurement plan  and use of cost reimbursement  contracts is very risky. We  recommended the program  report on the risks and  mitigation strategy for this  approach.
DOD agreed to report its  contracting strategy and plans  to Congress. In response to our  report recommendation, DOD  subsequently agreed to do a  schedule risk analysis, but still  had not done so as of February  2011. In February 2010, the  Department announced a  major restructuring of the JSF  program, including reduced  procurement and a planned  move to fixed-price contracts.
The program was restructured  to reflect findings of recent  independent cost team (JET II)  and independent  manufacturing review team. As  a result, development funds  increased, test aircraft were  added, the schedule was  extended, and the early  production rate decreased.
Because of additional costs  and schedule delays, the  program’s ability to meet  warfighter requirements on  time is at risk. We recommend  the program complete a full  comprehensive cost estimate  and assess warfighter and IOC  requirements. We suggest that  Congress require DOD to  prepare a “system maturity  matrix”–a tool for tying annual  procurement requests to  demonstrated progress.
DOD continued restructuring  actions and announced plans  to increase test resources and  lower the production rate.  Independent review teams  evaluated aircraft and engine  manufacturing processes. As  we projected in this report, cost  increases later resulted in a  Nunn-McCurdy breach. Military  services are currently reviewing  capability requirements as we  recommended. The  Department and Congress are  working on a “system maturity  matrix” tool, which we  suggested to Congress for  consideration, to improve  oversight and inform budget  deliberations.
Average procurement unit cost.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Acknowledgments
paragraphs: In addition to the contact name above, the following staff members made  key contributions to this report: Bruce Fairbairn, Assistant Director;  Charlie Shivers; Julie Hadley; Matt Lea; Jason Lee; Sean Merrill; LeAnna  Parkey; Karen Richey; Dr. W. Kendal Roberts; and Robert Swierczek.
section_title: Related GAO Products
paragraphs: Joint Strike Fighter: Restructuring Should Improve Outcomes, but  Progress Is Still Lagging Overall. GAO-11-450T. Washington, D.C.: March  15, 2011.
Tactical Aircraft: Air Force Fighter Force Structure Reports Generally  Addressed Congressional Mandates, but Reflected Dated Plans and  Guidance, and Limited Analyses. GAO-11-323R. Washington, D.C.:  February 24, 2011.
Defense Management: DOD Needs to Monitor and Assess Corrective  Actions Resulting from Its Corrosion Study of the F-35 Joint Strike  Fighter. GAO-11-171R. Washington D.C.: December 16, 2010.
Joint Strike Fighter: Assessment of DOD’s Funding Projection for the  F136 Alternate Engine. GAO-10-1020R. Washington, D.C.: September 15,  2010.
Tactical Aircraft: DOD’s Ability to Meet Future Requirements is  Uncertain, with Key Analyses Needed to Inform Upcoming Investment  Decisions. GAO-10-789. Washington, D.C.: July 29, 2010.
Defense Acquisitions: Assessments of Selected Weapon Programs.   GAO-10-388SP. Washington, D.C.: March 30, 2010.
Joint Strike Fighter: Significant Challenges and Decisions Ahead.   GAO-10-478T. Washington, D.C.: March 24, 2010.
Joint Strike Fighter: Additional Costs and Delays Risk Not Meeting  Warfighter Requirements on Time. GAO-10-382. Washington, D.C.: March  19, 2010.
Joint Strike Fighter: Significant Challenges Remain as DOD  Restructures Program. GAO-10-520T. Washington, D.C.: March 11, 2010.
Joint Strike Fighter: Strong Risk Management Essential as Program  Enters Most Challenging Phase. GAO-09-711T. Washington, D.C.: May 20,  2009.
Defense Acquisitions: Assessments of Selected Weapon Programs.   GAO-09-326SP. Washington, D.C.: March 30, 2009.
Joint Strike Fighter: Accelerating Procurement before Completing  Development Increases the Government's Financial Risk. GAO-09-303.  Washington D.C.: March 12, 2009.
Defense Acquisitions: Better Weapon Program Outcomes Require  Discipline, Accountability, and Fundamental Changes in the  Acquisition Environment. GAO-08-782T. Washington, D.C.: June 3, 2008.
Defense Acquisitions: Assessments of Selected Weapon Programs.   GAO-08-467SP. Washington, D.C.: March 31, 2008.
Joint Strike Fighter: Impact of Recent Decisions on Program Risks.  GAO-08-569T. Washington, D.C.: March 11, 2008.
Joint Strike Fighter: Recent Decisions by DOD Add to Program Risks.  GAO-08-388. Washington, D.C.: March 11, 2008.
Tactical Aircraft: DOD Needs a Joint and Integrated Investment  Strategy. GAO-07-415. Washington, D.C.: April 2, 2007.
Defense Acquisitions: Assessments of Selected Weapon Programs.   GAO-07-406SP. Washington, D.C.: March 30, 2007.
Defense Acquisitions: Analysis of Costs for the Joint Strike Fighter  Engine Program. GAO-07-656T. Washington, D.C.: March 22, 2007.
Joint Strike Fighter: Progress Made and Challenges Remain. GAO-07-360.  Washington, D.C.: March 15, 2007.
Tactical Aircraft: DOD’s Cancellation of the Joint Strike Fighter  Alternate Engine Program Was Not Based on a Comprehensive Analysis.  GAO-06-717R. Washington, D.C.: May 22, 2006.
Defense Acquisitions: Major Weapon Systems Continue to Experience  Cost and Schedule Problems under DOD’s Revised Policy. GAO-06-368.  Washington, D.C.: April 13, 2006.
Defense Acquisitions: Actions Needed to Get Better Results on Weapons  Systems Investments. GAO-06-585T. Washington, D.C.: April 5, 2006.
Tactical Aircraft: Recapitalization Goals Are Not Supported by  Knowledge-Based F-22A and JSF Business Cases. GAO-06-487T.  Washington, D.C.: March 16, 2006.
Joint Strike Fighter: DOD Plans to Enter Production before Testing  Demonstrates Acceptable Performance. GAO-06-356. Washington, D.C.:  March 15, 2006.
Joint Strike Fighter: Management of the Technology Transfer Process.  GAO-06-364. Washington, D.C.: March 14, 2006.
Tactical Aircraft: F/A-22 and JSF Acquisition Plans and Implications  for Tactical Aircraft Modernization. GAO-05-519T. Washington, D.C:  April 6, 2005.
Tactical Aircraft: Opportunity to Reduce Risks in the Joint Strike  Fighter Program with Different Acquisition Strategy. GAO-05-271.  Washington, D.C.: March 15, 2005.