Background

The Joint Strike Fighter Program is structured to use a common production line to produce three versions of a single aircraft. These aircraft will be tailored to meet conventional flight requirements for the U.S. Air Force, short take-off and vertical landing characteristics for the U.S. Marine Corps, and carrier operation suitability needs for the U.S. Navy. The program will also provide aircraft to the British Royal Navy and Air Force. Table 1 shows the services’ planned use for the Joint Strike Fighter.
A key objective of the Joint Strike Fighter acquisition strategy is affordability—reducing the development, production, and ownership costs of the program relative to prior fighter aircraft programs. To achieve its affordability objective, the Joint Strike Fighter program has incorporated various acquisition initiatives into the program’s acquisition strategy and various technological advances into the fighter. Among the acquisition initiatives planned was to develop critical technologies to a level where they represent low technical risk before the engineering and manufacturing contract is awarded. The expectation was that incorporating these initiatives into the acquisition strategy would avoid cost growth, schedule slippage, and performance shortfalls that have been experienced in other weapon acquisition programs.
To date, the Joint Strike Fighter Program has awarded contracts totaling over $2 billion to Boeing and Lockheed Martin for the current concept demonstration phase. During this phase, DOD required each contractor to design and build two aircraft to demonstrate the following:   commonality/modularity to validate the contractors’ ability to produce three aircraft versions on the same production line; the aircraft’s ability to do a short take-off and vertical landing, hover, and transition to forward flight; and   satisfactory low airspeed, carrier approach flying and handling qualities.
Each contractor was required to submit a Preferred Weapon System Concept, which outlines its final design concept for developing a Joint Strike Fighter aircraft that is affordable and meets performance requirements. The Preferred Weapon System Concept includes results from the flight and ground demonstrations and is being used by DOD to select the winning aircraft design and to award the engineering and manufacturing development contract. During engineering and manufacturing development, the Joint Strike Fighter will be fully developed, engineered, designed, fabricated, tested, and evaluated to demonstrate that the production aircraft will meet stated requirements. Critical junctures in engineering and manufacturing development are the preliminary and critical design reviews and commitments; testing of aircraft; and commitments to production hardware, including the purchase of long lead production items. It is at the critical design review that decisions are made toward finalizing the aircraft design and begin building test aircraft. About two-thirds of engineering and manufacturing development funding will be spent after this review. Figure 1 shows planned Joint Strike Fighter aircraft designs by contractor.

	Technology Readiness Assessments Provide Opportunities to Improve Outcomes

In our previous work on best business practices, commercial firms have told us that a key part of product development is getting the technology into the right size, weight, and configuration needed for the intended product—in this case, the final Joint Strike Fighter design. Once this has been demonstrated, the technology is at an acceptable level for product development. Technology readiness levels (TRL) can be used to assess the maturity of technology and can reveal whether a gap exists between a technology’s maturity and the maturity demanded for successful inclusion in the intended product. Defining this gap for the Joint Strike Fighter technologies is important for determining whether they can be expected to demonstrate required capabilities before being integrated into the aircraft design. Readiness levels are measured along a scale of one to nine, starting with paper studies of the basic concept, proceeding with laboratory demonstrations, and ending with a technology that has proven itself on the intended product. (See app. I for a detailed description of TRLs.) The Air Force Research Laboratory considers TRL 7 an acceptable risk for starting the engineering and manufacturing development phase. The readiness level definitions state that for a technology to be rated at TRL 7, it must be demonstrated using prototype hardware (such as a complete radar subsystem) that is the same size, weight, and configuration as that called for in the final aircraft design and that prototype has to be demonstrated to work in an environment similar to the planned operational system.
We have previously reviewed the impact of incorporating technologies into new product and weapon system designs. The results showed that programs met product objectives when the technologies were matured to higher levels and conversely showed that cost and schedule problems arose when programs started when technologies were at low readiness levels. For example, the Joint Direct Attack Munition (JDAM) used modified variants of proven components for guidance and global positioning. It also used mature, existing components from other proven manufacturing processes for its own system for controlling tail fin movements. The munition was touted for its performance in Kosovo and was purchased for less than half of its expected unit cost. However, the Comanche helicopter program began with critical technologies such as the engine, rotor, and integrated avionics at TRL levels of 5 or below. That program has seen 101 percent cost growth and 120-percent schedule slippage as a result of these low maturity levels and other factors.
In commenting on our report concerning better management of technology development, DOD agreed that TRLs are important and necessary in assisting decision makers in deciding on when and where to insert new technologies into weapons system programs and that it is desirable to mature technologies to TRL 7 prior to entering the engineering and manufacturing development phase of a weapon system program.Since that time, DOD has adopted the technology readiness levels as a means of assessing the technological maturity of new major programs. In a July 5, 2001, memorandum, the Deputy Under Secretary of Defense (Science and Technology) stated that new DOD regulations require that the military services’ science and technology executives conduct a technology readiness level assessment for critical technologies identified in major weapon systems programs prior to the start of engineering and manufacturing development and production. The memorandum notes that technology readiness levels are the preferred approach for all new major programs unless the Deputy Under Secretary approves an equivalent assessment method.
The Joint Strike Fighter Program, like many other DOD programs, has used risk management plans and engineering judgment as a way of assessing technological maturity. The Principal Deputy Under Secretary of Defense (Acquisition and Technology) has determined that these means will continue to be used by DOD and the Joint Strike Fighter contractors to assess the program’s technological risk. Risk management plans and judgment are necessary to managing any major development effort like the Joint Strike Fighter. However, without an underpinning such as technology readiness levels that allow transparency into program decisions, these methods allow significant technical unknowns to be judged acceptable risks because a plan exists for resolving the unknowns in the future. Experience on previous programs has shown that such methods have rarely assessed technical unknowns as a high or unacceptable risk; consequently, they failed to guide programs to meet promised outcomes. Technology readiness levels are based on actual demonstrations of how well technologies actually perform. Their strength lies in the fact that they characterize knowledge that exists rather than plans to gain knowledge in the future; they are, thus, less susceptible to optimism.

	Critical Technologies Are Not Projected to Be at Low Risk

In May 2000 we reported that all of the eight technologies identified by the Joint Strike Fighter program office as critical to the program were expected to be at maturity levels below that considered acceptable for low risk when entering engineering and manufacturing development (TRL 7). The eight critical technologies are: prognostics and health management, integrated flight propulsion control, subsystems, integrated support system, integrated core processor, radar, manufacturing, and mission systems integration. (See app. II for a description of these technologies.) During our review last year, we worked with the two competing contractors and the program office to arrive at the applicable TRLs for the critical technologies. Specifically, on separate visits to the contractors, with program office personnel present, we asked the contractors’ relevant technology managers to score the technologies they considered critical to enable their Joint Strike Fighter design to meet DOD requirements for the aircraft. At that time, we also asked them to describe their plans to mature the technologies to the planned start of the engineering and manufacturing development phase, then scheduled for April 2001. Upon reviewing these scores with the program office and in order to gain an overall Joint Strike Fighter Program perspective on technical maturity, the Joint Strike Fighter office agreed to provide us with TRL scores for the eight technologies they considered critical for meeting program cost and performance requirements. Figure 2 reflects the program office scores at the time of our last review. Due to the current Joint Strike Fighter competition, the specific technologies mentioned previously are not linked to scores so as not to divulge competition sensitive information.
As the figure shows, all eight technologies were projected to be below the level of maturity (TRL 7) considered acceptable for low risk when entering the engineering and manufacturing development phase and six of the technologies were projected to be below the level of maturity (TRL 6) that is considered low risk for entering the demonstration phase, which the Joint Strike Fighter Program began in 1996.
During our current review, we again visited the two competing contractors to discuss the status of the eight technologies. We learned that they have essentially accomplished, or plan to accomplish by October 2001, the technology development and demonstrations that they planned to accomplish as of April 2001. Thus figure 2 represents the current assessment of technical maturity. While two of these areas are very close to appropriate maturity levels, the Joint Strike Fighter’s critical technologies are not projected to be matured to levels that we believe would indicate a low risk program at the planned start of the engineering and manufacturing development phase. Key component technologies remain at higher risk levels for engineering and manufacturing development because (1) they have not been developed to approximately the same size, weight, and configuration called for in the final aircraft design and/or (2) they have not been demonstrated to work in an environment similar to the planned operational system.
The Joint Strike Fighter Program has made good progress in some technology areas. For example, contractor and program officials told us that because of concerns about propulsion technology, both contractors focused considerable attention on that area. Both contractors flew aircraft that demonstrated the capability for short take-off and vertical landing and accumulated at least 20 hours of flight time on those aircraft, which should satisfy the requirement in the Fiscal Year 2001 National Defense Authorization Act. In some other areas, the technology maturation has not been uniform across all critical components of a technology. For example, the radar has a number of critical components that must work together as a system. Both contractors have made considerable progress on one or more of those components, but the other critical components have not been matured to an acceptable level of risk. In order for this technology to achieve a TRL level of 7, all components had to be (1) demonstrated in the size and weight required to meet aircraft capabilities, (2) integrated together as they would be in the final aircraft design, and (3) flown in an environment similar to what the Joint Strike Fighter will be subjected. To demonstrate some critical technologies, both contractors flew key electronic and other components in flying avionics test beds (commercial aircraft reconfigured as flying laboratories). While these tests occurred in a relevant environment (e.g., in flight), the tested hardware was not always the same size and weight required for the Joint Strike Fighter aircraft. Conversely, some components were built to the required size and weight, but were demonstrated only in ground-testing environments. By not having matured all critical technology areas to appropriate maturity levels, the program remains at risk for achieving cost and performance goals upon entering product development.

	Conclusions

Moving into engineering and manufacturing development creates an expectation that the Joint Strike Fighter can be delivered for a stated time and dollar investment and with a given set of capabilities. The decisions the Department of Defense makes now and over the next 2 years will largely determine whether those expectations can be met.
A key component of the Joint Strike Fighter Program’s acquisition strategy is to enter the engineering and manufacturing development phase with low technical risk. The program will not have achieved that point by October 2001 because technologies, which the Joint Strike Fighter Program Office identified as critical to meeting the program’s cost and requirements objectives, will not have been matured to an acceptable risk level. By entering the engineering and manufacturing development phase with immature critical technologies, the program will need to continue to develop those technologies at the same time it will be concentrating on production issues and the integration of subsystems into a Joint Strike Fighter. This approach would not be consistent with best practices. In fact, it would more closely follow DOD’s traditional practices in weapon system programs that have often resulted in cost increases, schedule delays, and compromised performance.

	Recommendations for Executive Action

To eliminate one of the major sources of cost and schedule risk, we recommend that the Secretary of Defense delay the start of engineering and manufacturing development until critical technologies are matured to acceptable levels.
Alternatively, if the Secretary of Defense decides to accept these risks and move the program into engineering and manufacturing development as scheduled, we recommend that the Secretary dedicate the resources to ensuring that maturity of the critical technologies is demonstrated by the critical design review or defer the inclusion of immature technologies from the approved design.

	Agency Comments and Our Evaluation

In written comments on a draft of this report, the Director of Strategic and Tactical Systems, within the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, partially concurred with our recommendation.  DOD contended that an independent technology readiness assessment it carried out on the program showed that technology has extensively matured and the program is now ready to enter into systems development and demonstration.   DOD also stated that the Joint Strike Fighter Program Office has implemented a risk management program that will continue to monitor and address technology risks, as well as other risks, throughout the program’s life.  The full text of DOD’s comments is included in appendix III.
We disagree with the Department’s assessment of technological maturity. The TRL assessment conducted  as part of our review showed that technologies critical to the Joint Strike Fighter Program are not projected to be matured to levels that we believe would indicate a low risk program at the planned start of the engineering and manufacturing development phase.  Many of the technologies have not been demonstrated in their appropriate size and weight, nor have they been demonstated to function in an environment in which they will be used.  For example, many of the technologies are still in the laboratory and will require considerable maturation before they can be incorporated into the final design.  By entering the engineering and manufacturing development phase with immature critical technologies, the program will need to continue to develop those technologies at the same time it will be concentrating on engineering, designing, and fabricating the product.  As it has with many other DOD programs, this approach increases the likelihood of schedule delays and program cost increases. This is primarily why DOD’s new acquisition regulations emphasize separating technology development from product development.  In fact, experience has shown that resolving technology problems in product development can result in at least a ten- fold cost increase.
Moreover, DOD incorrectly states that the tools it used to assess its technology and the TRLs used for our review are equivalent methodologies for assessing technological maturity.  The Willoughby Templates used by DOD are a risk management tool.  They can be an excellent way to manage program risks, but in practice they have not been used to identify risk.  Identifying risk is the first step to managing it.  By contrast, by focusing specifically on assessing technology maturity against objective standards, TRLs have proven successful at identifying risks.  A more appropriate approach for DOD to take is to use technology readiness levels in conjunction with a management tool such as the Willoughby templates since this can result in more informed decision making and fewer unanticipated problems in an acquisition program. In fact, the Joint Strike Fighter program provides DOD with an excellent opportunity to apply these concepts in tandem.

	Scope and Methodology

To assess whether the Joint Strike Fighter’s critical technologies are projected to mature to low technical risk at the start of the engineering and manufacturing development phase, we used the technology readiness level tool and information provided by Joint Strike Fighter program officials and contractor officials at the Boeing Company, Seattle, Washington; Lockheed Martin Aeronautics Company, Fort Worth, Texas; and Pratt & Whitney, East Hartford, Connecticut. During our previous review, we had obtained detailed briefings from Boeing and Lockheed Martin officials on their plans to mature critical technologies prior to the date for awarding the engineering and manufacturing development contract, then scheduled for April 2001. We had also obtained program office and contractor assessments of the expected technology readiness levels for the critical technologies at April 2001. During our current review, we obtained detailed briefings from program office personnel on the status of critical technologies. We also obtained detailed briefings from Boeing, Lockheed Martin, and Pratt & Whitney officials on the contractors’ progress in maturing critical technologies and any further maturation plans through October 2001. We compared the latest information from the program office and the contractors to the information obtained during our prior review to determine if the critical technologies had been matured to higher technology readiness levels and the levels achieved.
We conducted our review from April through September 2001 in accordance with generally accepted government auditing standards.
We are sending copies of this report to the congressional defense committees; the Honorable Donald H. Rumsfeld, Secretary of Defense; the Honorable James G. Roche, Secretary of the Air Force; the Honorable Gordon R. England, Secretary of the Navy; General James L. Jones, Commandant of the Marine Corps; and the Honorable Mitchell E. Daniels, Jr., Director, Office of Management and Budget. We will also make copies available to other interested parties on request.
Please contact me at (202) 512-4841 if you or your staff have any questions concerning this report. Key contributors to this report were Robert Pelletier and Brian Mullins.

Appendix I: Technology Readiness Levels and Their Descriptions

	2. Technology concept and/or application formulated.

6. System/subsystem model or prototype demonstration in a relevant environment. 7. System prototype demonstration in an operational environment.
Description Lowest level of technology readiness. Scientific research begins to be translated into applied research and development. Examples might include paper studies of a technology’s basic properties. Invention begins. Once basic principles are observed, practical applications can be invented. The application is speculative and there is no proof or detailed analysis to support the assumption. Examples are still limited to paper studies. Active research and development is initiated. This includes analytical studies and laboratory studies to physically validate analytical predictions of separate elements of the technology. Examples include components that are not yet integrated or representative. Basic technological components are integrated to establish that the pieces will work together. This is relatively “low fidelity” compared to the eventual system. Examples include integration of “ad hoc” hardware in a laboratory. Fidelity of breadboard technology increases significantly. The basic technological components are integrated with reasonably realistic supporting elements so that the technology can be tested in a simulated environment. Examples include “high fidelity” laboratory integration of components. Representative model or prototype system, which is well beyond the breadboard tested for TRL 5, is tested in a relevant environment. Represents a major step up in a technology’s demonstrated readiness. Examples include testing a prototype in a high fidelity laboratory environment or in simulated operational environment. Prototype near or at planned operational system. Represents a major step up from TRL 6, requiring the demonstration of an actual system prototype in an operational environment, such as in an aircraft, vehicle, or space. Examples include testing the prototype in a test bed aircraft. Technology has been proven to work in its final form and under expected conditions. In almost all cases, this TRL represents the end of true system development. Examples include developmental test and evaluation of the system in its intended weapon system to determine if it meets design specifications. Actual application of the technology in its final form and under mission conditions, such as those encountered in operational test and evaluation. In almost all cases, this is the end of the last “bug fixing” aspects of true system development. Examples include using the system under operational mission conditions.

Appendix II: Critical Technologies and Their Descriptions

Appendix II: Critical Technologies and Their Descriptions Description Involves the ability to detect and isolate the cause of aircraft problems and then predict when maintenance activity will have to occur on systems with pending failures. Life-cycle cost savings are dependent on prognostics and health management through improved sortie generation rate, reduced logistics and manpower requirements, and more efficient inventory control. Includes integration of propulsion, vehicle management system, and other subsystems as they affect aircraft stability, control, and flying qualities (especially short take-off and vertical landing). Aircraft improvements are to reduce pilot workload and increase flight safety. Includes areas of electrical power, electrical wiring, environmental control systems, fire protection, fuel systems, hydraulics, landing gear systems, mechanisms, and secondary power. Important for reducing aircraft weight, decreasing maintenance cost, and improving reliability. Involves designing an integrated support concept that includes an aircraft with supportable stealth characteristics and improved logistics and maintenance functions. Life-cycle cost savings are expected from improved low observable maintenance techniques and streamlined logistics and inventory systems. Includes the ability to use commercial-based processors in an open architecture design to provide processing capability for radar, information management, communications, etc. Use of commercial processors reduces development and production costs and an open architecture design reduces future development and upgrade costs. Includes advanced integration of communication, navigation, and identification functions and electronic warfare functions through improved apertures, antennas, modules, radomes, etc. Important for reducing avionics cost and weight, and decreasing maintenance cost through improved reliability. Involves lean, automated, highly efficient aircraft fabrication and assembly techniques. Manufacturing costs should be less through improved flow time, lower manpower requirements, and reduced tooling cost. Involves decreasing pilot workload by providing information for targeting, situational awareness, and survivability through fusion of radar, electronic warfare, and communication, navigation, and identification data. Improvements are achieved through highly integrated concept of shared and managed resources, which reduces production costs, aircraft weight, and volume requirements, in addition to improved reliability.

Appendix III: Comments From the Department of Defense

Summary:
What GAO Found

The Joint Strike Fighter Program (JSFP), the military's most expensive aircraft program, is intended to produce affordable, next-generation aircraft to replace aging aircraft in military inventories. Although JSFP has made good progress in some technology areas, the program may not meet its affordability objective because critical technologies are not projected to be matured to levels GAO believes would indicate a low risk program at the planned start of engineering and manufacturing development in October 2001.