NASA's James Webb Space Telescope: Knowledge-Based Acquisition Approach Key to Addressing Program Challenges
Released: 2006-07-14T00:00:00

Letter

Background

The JWST—identified by the National Research Council as the top priority new initiative for astronomy and physics for the current decade—is a large deployable space-based observatory being developed to study and answer fundamental questions ranging from the formation and structure of the universe to the origin of planetary systems and the origins of life. Often referred to as the replacement to Hubble, the JWST is more of a next generation telescope—one that scientists believe will be capable of seeing back to the origins of the universe (Big Bang). The JWST will have a large, segmented primary mirror—6.5 meters (about 21 feet) in diameter—which is a leap ahead in technology over the last generation of mirrors. The observatory requires a sunshield approximately the size of a tennis court to allow it to cool to the extremely cold temperature (around 40 degrees Kelvin, or minus 388 degrees Fahrenheit) necessary for the telescope and science instruments to work. The mirror and the sunshield—both critical components—must fold up to fit inside the launch vehicle and open to their operational configuration once the JWST is in orbit. In addition, the observatory will house science instruments—such as a near-infrared camera, a near-infrared spectrograph, a mid-infrared instrument, and a fine guidance sensor—to enable scientists to conduct various research activities.

The JWST is an international collaboration among the United States, the European Space Agency (ESA), and the Canadian Space Agency (CSA). ESA will provide the near-infrared spectrograph science instrument, the optical bench assembly of the mid-infrared instrument, and the launch of the JWST by means of an Ariane 5 expendable launch vehicle. CSA’s contribution will be the fine guidance sensor to enable stable pointing.

Recently, the JWST program recognized significant cost growth and schedule slippage. In March 2005, NASA identified about $1 billion cost growth, which increased the JWST’s life-cycle cost estimate from $3.5 billion to $4.5 billion. In addition, the program’s schedule slipped nearly 2 years. As a result, the program began a series of re-baselining efforts to revise its acquisition strategy. In summer 2005, NASA Headquarters chartered two independent review teams—an Independent Review Team from NASA’s Independent Program Assessment Office and a Science Assessment Team—to evaluate the program. The Independent Review Team was charged with examining the program’s new cost/schedule/ technical baseline and reported in mid-April 2006 that (1) the JWST’s scientific performance met the expectations of the science community, (2) the technical content was complete and sound, and (3) the Goddard Space Flight Center and contractor teams were effective. However, the team was concerned about the program’s early year funding constraints.

The Science Assessment Team, an international team of outside experts, was established to evaluate scientific capabilities of the JWST in the 2015 time frame in light of other astronomical facilities that would be available. The team concluded that the financial savings gained from the reduction in the size of the primary mirror area would not be worth the resultant loss of scientific capabilities. The team recommended relaxing some science requirements and simplifying other aspects of the mission, such as integration and testing, to reduce the program’s cost risk. For example, the team recommended relaxing the contamination requirements, allowing the project to test the mirrors using an innovative approach that will reduce costs. The team also recommended that the JWST de-emphasize the shorter wavelengths, since other astronomical facilities would be available to cover that range.

JWST’s Revised Strategy Does Not Fully Incorporate a Knowledge-Based Approach That Could Reduce Risks and Better Inform Decision Making

The JWST program recently revised its acquisition strategy to conform to NASA’s acquisition policies; however, the program still faces considerable challenges. GAO best practices work has found that using a knowledge- based approach is a key factor in program success. When we initiated our work and before the program’s recently revised acquisition strategy, program officials intended to have NASA commit to the program and start implementation with immature technologies, according to best practices, and without a preliminary design. During our review, we discussed these shortfalls with NASA officials, and they revised their acquisition strategy to align their decision milestones in accordance with NASA acquisition policy. While this is a good step, the current strategy does not fully incorporate a knowledge-based approach that could reduce the program’s risks by ensuring that resources match requirements at program start. By closely following a knowledge-based approach, the JWST program will increase its chances for success and better inform NASA’s decision making.

Immature Technologies, Design Challenges, and Testing Restrictions Still Pose Risks

The JWST contains several innovations, including lightweight optics, a deployable sunshield, and a folding segmented mirror. Although the program began risk reduction activities early to develop and mature some technologies, such as the lightweight segmented folding mirror, the program is challenged with maturing some of its other critical technologies. For example, the sunshield, which consists of five layers of membranes, must be folded for launch but then unfurled to its operational configuration—with enough tension to prevent wrinkle patterns that could interfere with the telescope’s mirrors, but not so much tension to cause tears in the fabric. The sunshield must also be aligned with the rest of the observatory so that only the top layer of the sunshield is visible to the primary mirror and a correct angle between the observatory and the sun and other heat-radiating bodies is maintained to enable the telescope and science instruments to preserve the very cold temperature—about 40 degrees Kelvin—critical for achieving the JWST’s mission. In addition, using passive cooling devices, such as heat switches, to allow specific areas of the telescope to cool down, represent additional challenges since these items will be used in new configurations. NASA also recently substituted the cryo-cooler used for the mid-infrared instrument for a lower technology component to save mass. According to JWST officials, the program recently awarded the development contract for the cryo- cooler. In addition, the micro shutter array, which will allow the JWST to program specific patterns of the electromagnetic spectrum for viewing, is a new technology being developed by the Goddard Space Flight Center and is still at a relatively low level of maturity. JWST officials acknowledge that they are concerned about maturing the cryo-cooler and the micro shutter array.

In addition, the program also faces design challenges related to the launch vehicle and the observatory’s stability. For example, program officials told us that they may need to request a waiver because the telescope will not fit within the criteria limits of the launch vehicle’s envelop without making design modifications. Furthermore, due to the late selection of the launch vehicle, the project office and prime contractor are just beginning to discuss interfaces, transportation at the launch site, and the additional space issue with Ariane 5 officials. Also, the project faces the unresolved problem of finding the best way to keep the observatory stable. The large sunshield, observatory attitude changes, and other effects conspire to produce unbalanced torques, which can make the observatory unstable. The project continues to look at ways to resolve this problem, including thrusters to rebalance the observatory, but project officials say this will continue to be a challenge.

Another overriding concern is NASA’s inability to test the entire observatory in its operational environment, since there is no test facility in the United States large enough to perform this test. The plan is to incrementally test components and subsystems on the ground in laboratories simulating the observatory’s operational environment and to make extensive use of modeling and simulation. According to the memorandum summarizing the January 2006 System Definition Review, a key concern is that the JWST is pushing the limits of ground test facilities and cannot be tested at the observatory level; therefore, requiring complicated integration and testing with a series of subsystem tests and analyses. In its April 2006 assessment of the JWST program, the Independent Review Team reported that there are several exceptions to the “test as you fly” guideline and that mitigation strategies need to be developed before the end of the preliminary design phase.

Containing Further Cost Growth and Schedule Slippage

In March 2005, the JWST program recognized that its cost had grown by about $1 billion, increasing the JWST’s life-cycle cost estimate from $3.5 billion to $4.5 billion. About half of the cost growth was due to schedule slippage—a 1-year schedule slip because of a delay in the decision to use an ESA-supplied Ariane 5 launch vehicle and an additional 10-month slip caused by budget profile limitations in fiscal years 2006 and 2007. More than a third of the cost increase was caused by requirements and other changes. An increase in the program’s contingency funding accounted for the remainder—about 12 percent—of the growth.

Despite an increase in the program’s contingency funding, the Independent Review Team found that the contingency funding is still inadequate. In its April 2006 assessment of the JWST program’s re- baselining, the Independent Review Team expressed concern over the program’s contingency funding, stating that it is too low and phased in too late. According to the team, the program’s contingency from 2006 through 2010 of only $29 million, or about 1.5 percent, after “liens” and “threats” is inadequate. The team also stated that a 25 percent to 30 percent total contingency is appropriate for a program of this complexity. The program’s total contingency is only about 19 percent. The team warned that because of the inadequate contingency, the program’s ability to resolve issues, address program risk areas, and accommodate unknown problems is very limited. Therefore, the team concluded that from a budget perspective, the re-baselined program is not viable for a 2013 launch. The team recommended that before the Non-Advocate Review (NAR) leading to program start, steps should be taken by the Science Mission Directorate to assure that the JWST program contains an adequate time-phased funding contingency to secure a stable launch date.

The JWST program remains at risk of incurring additional cost growth and schedule slippage because of the technical challenges that must be resolved—immature technologies, design challenges, and testing restrictions. Our best practices work indicates that immature technology increases the risk of cost increases and schedule slips. Unresolved technology challenges can cascade through a product development cycle often resulting in an unstable design that will require more testing and thus more time and money to fix the problems. Subsequently, it will be difficult to prepare a reliable cost estimate until these challenges are resolved.

Knowledge-Based Approach Key to Overcoming Challenges

Our past work on the best practices of product developers in government and industry has found that the use of a knowledge-based approach is a key factor in successfully addressing challenges such as those faced by the JWST program. Over the last several years, we have undertaken a body of work on how leading developers in industry and government use a knowledge-based approach to deliver high quality products on time and within budget. A knowledge-based approach to product development efforts enables developers to be reasonably certain that, at critical junctures or “knowledge points” in the acquisition life cycle, their products are more likely to meet established cost, schedule, and performance baselines and therefore provides them with information needed to make sound investment decisions. The marker for the first juncture—knowledge point 1 (KP1)—occurs just prior to program start. At KP1, the customer’s requirements match the product developer’s resources in terms of knowledge, time, and money. At KP 2, the product design is stable, and production processes are mature at KP 3. Product development efforts that have not followed a knowledge-based approach can frequently be characterized by poor cost, schedule, and performance outcomes.

We recently reported that NASA’s revised acquisition policy for developing flight systems and ground support projects incorporates some aspects of the best practices used by successful developers. For example, NASA policy requires projects to conduct a major decision review—NAR— before moving from formulation to implementation. Further, before moving from formulation to implementation, projects must validate requirements and develop realistic cost and schedule estimates, human capital plans, a preliminary design, and a technology plan—all key elements for matching needs to resources before commitment to a major investment is made at project start. Figure 2 compares NASA’s life cycle with a knowledge-based acquisition life cycle.

NAR (KP1)

Knowledge Based Approach

Management Decision Reviews

While the policy incorporates elements of a knowledge-based approach, we also reported that NASA’s acquisition policies lack the necessary requirements to ensure that programs proceed and are funded only after an adequate level of knowledge at key junctures. For example, NASA policy does not require that programs demonstrate technologies at high levels of maturity at program start. Further, although NASA policy does require project managers to establish a continuum of technical and management reviews, the policy does not specify what these reviews should be nor does it require major decision reviews at other key points in a product’s development. These best practices could be used to further reduce program risks.

In order to close the gaps between NASA’s current acquisition environment and best practices on knowledge-based acquisition, we recommended that NASA take steps to ensure that NASA projects follow a knowledge-based approach for product development. Specifically, we recommended that NASA (1) in drafting its systems engineering policy, incorporate requirements for flight systems and ground support projects to capture specific product knowledge by key junctures in project development and use demonstration of this knowledge as exit criteria for decision making at key milestones and (2) revise NASA Procedural Requirements 7120.5C to institute additional major decision reviews following the NAR for flight systems and ground support projects, which result in recommendations to the appropriate decision authority at key milestones. NASA concurred with our recommendations and agreed to revise its policies.

One of the resources needed at program start is mature technology. Our best practices work has shown that technology readiness levels (TRL)— a concept developed by NASA—can be used to gauge the maturity of individual technologies. Specifically, TRL 6—demonstrating a technology as a fully integrated prototype in a realistic environment—is the level of maturity needed to minimize risks for space systems entering product development. To achieve TRL 6, technology maturity must be demonstrated in a relevant environment using a prototype or model. (See app. II for a detailed description and definition of TRLs and test environments.)

A knowledge-based approach also involves the use of incremental markers to ensure that the required knowledge has been attained at each critical juncture. For example, exit criteria at KP1 should include demonstrated maturity of critical technologies, completed trade-offs and finalized requirements, and initial cost and schedule estimates using results from the preliminary design review. The approach ensures that managers will (1) conduct activities to capture relevant product development knowledge, (2) provide evidence that knowledge was captured, and (3) hold decision reviews to determine that appropriate knowledge was captured to allow a move to the next phase. If the knowledge attained at each juncture does not justify the initial investment, the project should not go forward and additional resources should not be committed.

Risks Not Fully Addressed by Recently Revised Acquisition Strategy

Prior to the program’s recent acquisition strategy revision, program officials were not following NASA acquisition policy and were set to commit to the program and start implementation with immature technologies, according to best practices, and without a preliminary design. For instance, the schedule called for convening the NAR before the end of preliminary design. NASA policy indicates that the NAR and Preliminary Design Review (PDR) should be aligned. Even at the pre- NAR in July 2003, the plan had been to have the NAR before the PDR, although the two reviews were closer together than the more recent plan.

During our review, we discussed these shortfalls with NASA officials. To their credit, they revised their acquisition strategy to conform to NASA policy. Currently, the mission NAR—upon which the program start decision will be based—will be aligned with the mission PDR (scheduled for March 2008). We believe this is a positive step, since it will ensure that a preliminary design—a key element for matching needs to resources—is established before program start. The revised strategy also splits the NAR into two parts—a technical NAR and a mission NAR. The purpose of the technical NAR (scheduled for January 2007) will be to determine whether the project has successfully retired its invention risk, i.e., critical technologies have achieved TRL 6, according to a NASA official. Technology issues will not be revisited after the technical NAR unless problems arise. However, it is unclear if the critical technologies will be demonstrated to a level of fidelity required by best practices at the technical NAR. Furthermore, the strategy does not fully incorporate a knowledge-based approach that could address the program’s risks by ensuring—through the use of exit criteria—that resources match requirements in terms of knowledge, time, and money before program start. For example: Under a knowledge-based approach, adequate testing is required to demonstrate that key technologies are mature—at TRL 6—prior to program start. This is particularly important for the JWST, given the program’s challenges with testing restrictions and the fact that the observatory cannot be serviced in space. In some cases, such as the sunshield, backup technologies do not exist, thus increasing the importance of adequately maturing and testing critical technologies. If key components—like the sunshield—fail, then the entire observatory will be lost. This requires greater fidelity in the testing, even as early as demonstrating the maturity of key technologies prior to program start.

To achieve TRL 6 (the maturity level required by best practices for program start), technology maturity must be demonstrated as a representative model or prototype—which is very close to the actual system in form, fit, and function—in a relevant environment. However, there is risk that the current JWST technology development plan will not result in the appropriate demonstration of technology maturity. For example, the half-scale thermal vacuum test of the entire observatory at Johnson Space Center is currently planned for September 2008, and so the knowledge gained regarding the maturity of the sunshield’s thermal and dynamic performance is pushed out 6 months beyond the PDR/NAR/program start date of March 2008. When JWST program officials briefed us in August 2005, the TRL levels for thermal and dynamic performance of the sunshield were both assessed to be at TRL 4, and the plan to get to TRL 6 was to test these subsystems during this half-scale thermal vacuum test. However, in fall 2005 program officials reviewed the technology development plan and concluded that only the materials for the sunshield’s membrane are technology development items, while other items affecting the configuration and deployment of the sunshield—such as thermal and dynamic performance—are considered engineering challenges. JWST officials stated that earlier testing of sample materials demonstrated the sunshield’s thermal performance and a demonstration using a 1/10th scale model demonstrated dynamic performance and satisfied TRL 6 requirements. However, we have found in our best practices work that demonstrating a technology to a TRL 6 typically involves demonstrating that a prototype—close to the form, fit, and functionality intended for the product—has been demonstrated in an environment that closely represents the anticipated operational environment. In our past review of development programs, we have found that if this level of maturity is not demonstrated before a product development effort is launched, a program increases the likelihood of cost growth and schedule delays as it tries to close the knowledge gap between the technologies’ maturity level and the product’s design requirements.

The JWST program’s inadequate contingency runs contrary to another premise of a knowledge-based approach—having sufficient resources in terms of funding available to ensure a program’s success. As discussed in an earlier section, the Independent Review Team stated that the program’s contingency from 2006 through 2010 of only about 1.5 percent after “liens” and “threats” is inadequate. The team warned that, because of the inadequate contingency, the program’s ability to resolve issues, address program risk areas, and accommodate unknown problems is very limited. The team concluded that, from a budget perspective, the re-baselined program is not viable for a 2013 launch.

Knowledge-Based Approach Would Allow the JWST Program to Better Inform NASA’s Decision- Making Process

A good basis for making informed investment decisions is essential in the fiscally constrained environment that now exists across the federal government. Our nation faces large, growing, and structural long-term fiscal imbalances. Given the severity of those fiscal challenges and the wide range of federal programs, hard choices need to be considered across the government, and NASA is no exception. NASA must compete with other departments and agencies for part of a constricted discretionary spending budget.

In the near future, NASA will need to determine the resources necessary to develop the systems and supporting technologies to achieve the President’s Vision for Space Exploration—while simultaneously financing its other priority programs—and structure its investment strategy accordingly. Initial implementation of the Vision as explained in NASA’s Exploration Systems Architecture Study calls for completing the International Space Station, developing a new crew exploration vehicle, and returning to the moon no later than 2020. NASA estimates that it will cost approximately $104 billion over the next 13 years to accomplish these initial goals. These priorities, along with NASA’s other missions, will be competing within NASA for funding. It will likely be difficult for decision makers to agree on which projects to invest in and which projects, if any, to terminate. The NASA Administrator has acknowledged that NASA faces difficult choices about its missions in the future—for example, between human space flight, science, and aeronautics missions.

In the President’s fiscal year 2007 budget request for NASA, the JWST has the largest budget allocation of all programs in the Science Mission Directorate’s Astrophysics Division for the 5-year budget horizon from fiscal year 2007 through fiscal year 2011—nearly $2 billion of the division’s $6.9 billion total budget, or about 29 percent. An inadequately informed decision to commit to the estimated $4.5 billion total funding for the JWST would significantly impact NASA’s science portfolio, since funding given to the JWST will not available for other programs. Early in the planning for how to handle the JWST program’s cost growth, NASA officials recognized the impact that the JWST’s cost growth could have on other programs. In a July 2005 briefing to the Agency Program Management Council soon after the cost growth was identified, NASA officials stated that “something must give if JWST stays in the portfolio.” The choices discussed were (1) relaxing requirements or (2) adding budget and schedule, which would mean that other missions would be deferred or deleted from the portfolio.

In addition, committing to the JWST program obligates the government contractually, since it allows the prime contractor to begin implementation tasks on the very long prime contract extending from October 2002 through launch—currently planned for June 2013—plus one year. The contract states that until the project achieves the implementation milestone, contract spending is limited to formulation activities, except for long-lead items and other activities approved in writing. After the implementation milestone is achieved at program start, the contracting officer will notify the contractor by letter to proceed to implementation. According to the contracting officer, the assumption is that this is the go-ahead for the whole program.

To make well-informed decisions, NASA needs the knowledge to assess the value of its programs—like the JWST program—in relationship to each other. In May 2004, we reported that, of 27 NASA programs we examined, 17 had cost increases averaging about 31percent. One of the programs in our sample was another infrared telescope program—the Spitzer Space Telescope—and it was plagued by schedule slippages caused by delays in the delivery of components, flight software, the mission operation system, and launch delays, all contributing to a 29.3 percent increase in program costs. In general, we found the programs in the sample lacked sufficient knowledge needed to make informed acquisition decisions. Insufficient knowledge to make informed investment decisions can further complicate the already-difficult choices that NASA faces. Conversely, sufficient knowledge at key junctures can facilitate well-informed investment decisions and protect the government from incurring contractual liabilities before it is appropriate. A knowledge-based approach ensures that comprehensive and comparable programmatic data are obtained.

Conclusions

Within the JWST program, NASA officials have accomplished a great deal, such as the development of the large, segmented mirror that is a leap ahead in technology. Moreover, the program has support from the larger scientific community. To enhance the program’s chances for success, program officials have chosen a path forward which follows NASA’s policies for ensuring readiness to proceed into implementation/product development. However, the JWST program’s revised strategy does not fully address the risks associated with the many challenges that the program still faces—including maturing technology, mitigating testing restrictions, and ensuring that adequate funding is available for contingencies. This puts the program at risk of further cost growth and schedule slippage. The program needs to have sufficient knowledge at key junctures to successfully address its challenges and use incremental markers to make certain that resources in terms of knowledge, time, workforce, and money match the requirements. Given the severity of the fiscal challenges our nation faces and the wide range of competing federal programs, hard choices need to be considered across the government, and NASA is no exception. Using a knowledge-based approach for NASA’s new development programs such as the JWST could help the agency make the difficult choices about how to allocate its limited budget resources among competing priorities by utilizing common and consistent criteria in program evaluations.

Recommendations for Executive Action

To increase the JWST program’s chances of successful product development, we recommend that the NASA Administrator take the following actions:

Direct the JWST program to fully apply a knowledge-based acquisition approach—to include incremental markers—that will not only ensure that adequate knowledge is attained at key decision points, but also hold the program accountable. These markers should include, but not be limited to schedules that demonstrate the maturity of all critical technologies prior to program start; criteria to ensure the validity of test articles; criteria to demonstrate that mature component designs being used in new configurations meet form, fit, and function standards; and criteria to ensure that sufficient contingency funding can be provided and phased appropriately.

Instruct the JWST program to continue to adhere to NASA acquisition policy and base the program’s go/no-go review (NAR) decision not only on adherence to that policy, but also on (1) the program’s ability to demonstrate whether it is meeting the knowledge markers outlined earlier and (2) whether adequate funds are available to execute the program.

Agency Comments and Our Evaluation

In written comments on a draft of this report, NASA concurred with our two recommendations and outlined actions that the agency plans to take to implement such recommendations. NASA said that it endorses the knowledge-based approach recommended and that it believes the current JWST program plan is consistent with that approach. NASA’s recognition of the value of obtaining knowledge prior to moving to subsequent acquisition phases and acknowledgment that it plans to use exit criteria as knowledge markers for other JWST mission-level reviews are welcome steps toward establishing an agency-wide risk reduction culture. Now, it will be critical for NASA decision makers to enforce adherence to the discipline of the knowledge-based approach and ensure that critical product knowledge is indeed demonstrated before allowing the JWST program to proceed. In the years ahead, NASA decision makers will likely face pressures to grant waivers for going forward with immature technologies, allow programs to be restructured, and thus marginalize accountability. For a program such as the JWST, whose investment is already substantial and successful outcome eagerly anticipated by the science community, adherence to such knowledge-based principles will need to be strictly enforced. As identified in this report, NASA would be well served by applying its own technology readiness standards (reprinted in appendix II) as part of its exit criteria, and demonstrating that critical technologies are at the TRL 6 level prior to program start using a representative model or prototype—which is very close to the actual system in form, fit, and function—in a relevant environment. Emphasis by decision makers on the application of “form, fit, and function standards” and “validity of test articles” as exit criteria for the JWST program start and entry into Phase C will help address our concern that the current JWST technology development plan may not result in the appropriate demonstration of technology maturity prior to program start. NASA’s comments are reprinted in appendix III.

We are sending copies of this report to interested congressional committees and to the NASA Administrator. We will make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or lia@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are acknowledged in appendix IV.

Appendix I: Scope and Methodology

To assess the extent to which the JWST acquisition strategy follows NASA policy and GAO best practices for ensuring readiness to proceed into implementation, we reviewed NASA policy on program management and compared the JWST project office’s management approach to NASA policy. Additionally, we analyzed the JWST acquisition strategy and benchmarked it to best practices. We interviewed NASA and contractor officials to clarify our understanding of the JWST management approach and technology development plan in relation to NASA policy and guidelines and best practices. To deepen our understanding of JWST technical issues, we attended the 3-day Sunshield Subsystem Concept Design Review as well as the 4-day JWST System Definition Review.

To evaluate the impact of the JWST acquisition strategy on NASA’s ability to assess the program and make informed investment decisions in the context of its other priorities, we analyzed available JWST cost and schedule data and conducted interviews with program officials to clarify our understanding of the information. Furthermore, we requested and reviewed documentary support breaking out the components of the cost increases and schedule slippage. We also interviewed program officials to clarify our understanding of the potential impact that investment in the JWST will have on other NASA programs. In addition, we reviewed statements of the NASA Administrator, budget documents, GAO’s High- Risk Series, and GAO’s 21st Century Challenges to better evaluate the JWST’s significance in the larger NASA and federal government context.

To accomplish our work, we visited NASA Headquarters, Washington, D.C.; Goddard Space Flight Center, Greenbelt, Maryland; Marshall Space Flight Center, Huntsville, Alabama; Northrop Grumman Space Technology, Redondo Beach, California; and Ball Aerospace and Technologies Corporation, Boulder, Colorado.

We performed our review from August 2005 through May 2006 in accordance with generally accepted government auditing standards.

Appendix II: Technology Readiness Levels

Lowest level of technology readiness. Scientific research begins to be translated into applied research and development. Examples might include paper studies of a technology’s basic properties.

None. (Paper studies and analysis.)

Invention begins. Once basic principles are observed, practical applications can be invented. The application is speculative and there is no proof or detailed analysis to support the assumption. Examples are still limited to paper studies.

None. (Paper studies and analysis.)

Active research and development is initiated. This includes analytical studies and laboratory studies to physically validate analytical predictions of separate elements of the technology. Examples include components that are not yet integrated or representative.

Analytical studies and demonstration of nonscale individual components (pieces of subsystem).

Basic technological components are integrated to establish that the pieces will work together. This is relatively “low fidelity” compared to the eventual system. Examples include integration of “ad hoc” hardware in a laboratory.

Low fidelity breadboard. Integration of nonscale components to show pieces will work together. Not fully functional or form or fit but representative of technically feasible approach suitable for flight articles.

Fidelity of breadboard technology increases significantly. The basic technological components are integrated with reasonably realistic supporting elements so that the technology can be tested in a simulated environment. Examples include “high fidelity” laboratory Integration of components.

High fidelity breadboard. Functionally equivalent but not necessarily form and/or fit (size weight, materials, etc.). Should be approaching appropriate scale. May include integration of several components with reasonably realistic support elements/subsystems to demonstrate functionality.

Lab demonstrating functionality but not form and fit. May include flight demonstrating breadboard in surrogate aircraft. Technology ready for detailed design studies.

Representative model or prototype system, which is well beyond the breadboard tested for TRL 5, is tested in a relevant environment. Represents a major step up in a technology’s demonstrated readiness. Examples include testing a prototype in a high fidelity laboratory environment or in simulated operational environment.

Prototype—Should be very close to form, fit and function. Probably includes the integration of many new components and realistic supporting elements/subsystems if needed to demonstrate full functionality of the subsystem.

High-fidelity lab demonstration or limited/restricted flight demonstration for a relevant environment. Integration of technology is well defined.

Prototype near or at planned operational system. Represents a major step up from TRL 6, requiring the demonstration of an actual system prototype in an operational environment, such as in an aircraft, vehicle or space. Examples include testing the prototype in a test bed aircraft.

Prototype. Should be form, fit and function integrated with other key supporting elements/subsystems to demonstrate full functionality of subsystem.

Flight demonstration in representative operational environment such as flying test bed or demonstrator aircraft. Technology is well substantiated with test data.

Technology has been proven to work in its final form and under expected conditions. In almost all cases, this TRL represents the end of true system development. Examples include developmental test and evaluation of the system in its intended weapon system to determine if it meets design specifications.

Developmental test and evaluation in the actual system application.

Actual application of the technology in its final form and under mission conditions, such as those encountered in operational test and evaluation. In almost all cases, this is the end of the last “bug fixing” aspects of true system development. Examples include using the system under operational mission conditions.

Operational test and evaluation in operational mission conditions.

Appendix III: Comments from the National Aeronautics and Space Administration

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, Jim Morrison, Assistant Director; Greg Campbell; Keith Rhodes; Sylvia Schatz; Erin Schoening; Hai Tran; and Ruthie Williamson made key contributions to this report.

Related GAO Products

NASA: Implementing a Knowledge-Based Acquisition Framework Could Lead to Better Investment Decisions and Project Outcomes. GAO-06-218. Washington, D.C.: December 21, 2005.

NASA’s Space Vision: Business Case for Prometheus 1 Needed to Ensure Requirements Match Available Resources. GAO-05-242. Washington, D.C.: February 28, 2005.

Space Reports

Space Acquisitions: Stronger Development Practices and Investment Planning Need to Address Continuing Problems. GAO-05-891T. Washington, D.C.: July 12, 2005.

Defense Acquisitions: Incentives and Pressures That Drive Problems Affecting Satellite and Related Acquisitions. GAO-05-570R. Washington, D.C.: June 23, 2005.

Defense Acquisitions: Space-Based Radar Effort Needs Additional Knowledge before Starting Development. GAO-04-759. Washington, D.C.: July 23, 2004.

Defense Acquisitions: Risks Posed by DOD’s New Space Systems Acquisition Policy. GAO-04-379R. Washington, D.C.: January 29, 2004.

Space Acquisitions: Committing Prematurely to the Transformational Satellite Program Elevates Risks for Poor Cost, Schedule, and Performance Outcomes. GAO-04-71R. Washington, D.C.: December 4, 2003.

Defense Acquisitions: Improvements Needed in Space Systems Acquisition Policy to Optimize Growing Investment in Space. GAO-04-253T. Washington, D.C.: November 18, 2003.

Defense Acquisitions: Despite Restructuring, SBIRS High Program Remains at Risk of Cost and Schedule Overruns. GAO-04-48. Washington, D.C.: October 31, 2003.

Defense Acquisitions: Improvements Needed in Space Systems Acquisition Management Policy. GAO-03-1073. Washington, D.C.: September 15, 2003.

Military Space Operations: Common Problems and Their Effects on Satellite and Related Acquisitions. GAO-03-825R. Washington, D.C.: June 2, 2003.

Military Space Operations: Planning, Funding, and Acquisition Challenges Facing Efforts to Strengthen Space Control. GAO-02-738. Washington, D.C.: September 23, 2002.

Polar-Orbiting Environmental Satellites: Status, Plans, and Future Data Management Challenges. GAO-02-684T. Washington, D.C.: July 24, 2002.

Defense Acquisitions: Space-Based Infrared System-Low at Risk of Missing Initial Deployment Date. GAO-01-6. Washington, D.C.: February 28, 2001.

Best Practices Reports

Defense Acquisitions: Assessments of Selected Major Weapon Programs. GAO-05-301. Washington, D.C.: March 31, 2005.

Defense Acquisitions: Stronger Management Practices Are Needed to Improve DOD’s Software-Intensive Weapon Acquisitions. GAO-04-393. Washington, D.C.: March 1, 2004.

Defense Acquisitions: Assessments of Selected Major Weapon Programs. GAO-04-248. Washington, D.C.: March 31, 2004.

Defense Acquisitions: DOD’s Revised Policy Emphasizes Best Practices, but More Controls Are Needed. GAO-04-53. Washington, D.C.: November 10, 2003.

Defense Acquisitions: Assessments of Selected Major Weapon Programs. GAO-03-476. Washington, D.C.: May 15, 2003.

Best Practices: Setting Requirements Differently Could Reduce Weapon Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February 11, 2003.

Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 2002.

Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.

Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 2001.

Best Practices: A More Constructive Test Approach Is Key to Better Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 2000.

Defense Acquisition: Employing Best Practices Can Shape Better Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000.

Best Practices: DOD Training Can Do More to Help Weapon System Program Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 16, 1999.

Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 1999.

Defense Acquisitions: Best Commercial Practices Can Improve Program Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.

Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T-NSIAD-98-123. Washington, D.C.: March 18, 1998.

Best Practices: Successful Application to Weapon Acquisition Requires Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.: February 24, 1998.

Major Acquisitions: Significant Changes Underway in DOD’s Earned Value Management Process. GAO/NSIAD-97-108. Washington, D.C.: May 5, 1997.

Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 1996.