NASA's James Webb Space Telescope: Knowledge-Based Acquisition Approach Key to Addressing Program Challenges
Published: 2006-07-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The National Aeronautics and Space Administration's (NASA) James Webb Space Telescope (JWST) is being designed to explore the origins and nature of the universe. It should allow scientists to look deeper into space--and thus farther back in time--than ever before. The program, however, has experienced cost growth of more than $1 billion and its schedule has slipped nearly 2 years. NASA recently restructured the program and now anticipates a launch no sooner than June 2013. Because of the cost and schedule problems, under the Comptroller General's authority, we reviewed the JWST program to determine the extent to which this procurement follows NASA acquisition policy and GAO best practices for ensuring that adequate product knowledge is used to make informed investment decisions
section_title: What GAO Found
paragraphs: Although the JWST program recently revised its acquisition strategy to conform to NASA's acquisition policies, the program still faces considerable challenges because it has not fully implemented a knowledge-based approach, which our past work has shown is often a key factor in program success. In a recent report, we made recommendations that NASA take steps to ensure that projects follow a knowledge-based approach for product development. NASA concurred and revised its acquisition policy. When we initiated our work and before the JWST program's recently revised acquisition strategy, program officials intended to have NASA commit to program start, which is the end of the formulation phase and the beginning of the implementation phase, with immature technologies, according to best practices, and without a preliminary design. During our review, we discussed these shortfalls with NASA officials, and they revised their acquisition strategy to conform to NASA policy. However, the current strategy still does not fully incorporate a knowledge-based approach which ensures that resources match requirements in terms of knowledge, time, and money before program start. If program officials follow the current plan, the maturity of key technologies may not be adequately tested prior to program start. In addition, it appears the program will not have sufficient funding resources to ensure the program's success. In light of the fiscally constrained environment the federal government and NASA will face in the years ahead, adopting a knowledge-based approach will not only increase the JWST program's chances for success but also lay the foundation for comparison between competing programs.

section_title: Letter
paragraphs: As the expected follow-on to the tremendously successful Hubble Space  Telescope, the National Aeronautics and Space Administration’s (NASA)  James Webb Space Telescope (JWST) is being designed to explore the  early universe and allow scientists to shed light on the origins and nature  of the universe by allowing them to look deeper into space—and thus  farther back in time—than ever before. Recently, however, NASA  acknowledged that the program has experienced cost growth exceeding  $1 billion—which increased its life-cycle cost estimate from $3.5 billion to  $4.5 billion—and its schedule has slipped nearly 2 years. The agency  restructured the program and is now anticipating a launch no sooner than  June 2013.
Because of the restructuring and past cost and schedule problems, we  reviewed the program to determine the extent to which the JWST  program’s acquisition strategy follows NASA acquisition policy and  Government Accountability Office (GAO) best practices for ensuring that  adequate product knowledge is used to make informed investments. We  conducted our work under the Comptroller General’s authority and are  addressing this report to you because of your committee’s or  subcommittee’s interest in NASA activities.
To assess the extent to which the JWST acquisition strategy follows NASA  policy and GAO best practices for ensuring readiness to proceed into  implementation, we reviewed NASA policy guidance and compared it with  the JWST program’s acquisition strategy. We also benchmarked the JWST  acquisition strategy to best practices. We interviewed NASA and  contractor officials to clarify our understanding of the program’s  management approach and technology development plan. We analyzed  cost and schedule information and discussed the impact of the investment  in the JWST on other NASA programs with NASA officials. We attended  two design reviews, including one at the prime contractor’s facility. We  performed our review from August 2005 through May 2006 in accordance  with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: The JWST—identified by the National Research Council as the top priority  new initiative for astronomy and physics for the current decade—is a large  deployable space-based observatory being developed to study and answer  fundamental questions ranging from the formation and structure of the  universe to the origin of planetary systems and the origins of life. Often  referred to as the replacement to Hubble, the JWST is more of a next  generation telescope—one that scientists believe will be capable of seeing  back to the origins of the universe (Big Bang). The JWST will have a large,  segmented primary mirror—6.5 meters (about 21 feet) in diameter—which  is a leap ahead in technology over the last generation of mirrors. The  observatory requires a sunshield approximately the size of a tennis court  to allow it to cool to the extremely cold temperature (around 40 degrees  Kelvin, or minus 388 degrees Fahrenheit) necessary for the telescope and  science instruments to work. The mirror and the sunshield—both critical  components—must fold up to fit inside the launch vehicle and open to  their operational configuration once the JWST is in orbit. In addition, the  observatory will house science instruments—such as a near-infrared  camera, a near-infrared spectrograph, a mid-infrared instrument, and a  fine guidance sensor—to enable scientists to conduct various research  activities.
The JWST is an international collaboration among the United States, the  European Space Agency (ESA), and the Canadian Space Agency (CSA).  ESA will provide the near-infrared spectrograph science instrument, the  optical bench assembly of the mid-infrared instrument, and the launch of  the JWST by means of an Ariane 5 expendable launch vehicle. CSA’s  contribution will be the fine guidance sensor to enable stable pointing.
Recently, the JWST program recognized significant cost growth and  schedule slippage. In March 2005, NASA identified about $1 billion cost  growth, which increased the JWST’s life-cycle cost estimate from   $3.5 billion to $4.5 billion. In addition, the program’s schedule slipped  nearly 2 years. As a result, the program began a series of re-baselining  efforts to revise its acquisition strategy. In summer 2005, NASA  Headquarters chartered two independent review teams—an Independent  Review Team from NASA’s Independent Program Assessment Office and a  Science Assessment Team—to evaluate the program. The Independent  Review Team was charged with examining the program’s new  cost/schedule/ technical baseline and reported in mid-April 2006 that   (1) the JWST’s scientific performance met the expectations of the science  community, (2) the technical content was complete and sound, and (3) the  Goddard Space Flight Center and contractor teams were effective.  However, the team was concerned about the program’s early year funding  constraints.
The Science Assessment Team, an international team of outside experts,  was established to evaluate scientific capabilities of the JWST in the 2015  time frame in light of other astronomical facilities that would be available.  The team concluded that the financial savings gained from the reduction in  the size of the primary mirror area would not be worth the resultant loss of  scientific capabilities. The team recommended relaxing some science  requirements and simplifying other aspects of the mission, such as  integration and testing, to reduce the program’s cost risk. For example, the  team recommended relaxing the contamination requirements, allowing the  project to test the mirrors using an innovative approach that will reduce  costs. The team also recommended that the JWST de-emphasize the  shorter wavelengths, since other astronomical facilities would be available  to cover that range.
section_title: JWST’s Revised Strategy Does Not Fully Incorporate a Knowledge-Based Approach That Could Reduce Risks and Better Inform Decision Making
paragraphs: The JWST program recently revised its acquisition strategy to conform to  NASA’s acquisition policies; however, the program still faces considerable  challenges. GAO best practices work has found that using a knowledge- based approach is a key factor in program success. When we initiated our  work and before the program’s recently revised acquisition strategy,  program officials intended to have NASA commit to the program and start  implementation with immature technologies, according to best practices,  and without a preliminary design. During our review, we discussed these  shortfalls with NASA officials, and they revised their acquisition strategy  to align their decision milestones in accordance with NASA acquisition  policy. While this is a good step, the current strategy does not fully  incorporate a knowledge-based approach that could reduce the program’s  risks by ensuring that resources match requirements at program start. By  closely following a knowledge-based approach, the JWST program will  increase its chances for success and better inform NASA’s decision  making.
subsections: section_title: Immature Technologies, Design Challenges, and Testing Restrictions Still Pose Risks
paragraphs: The JWST contains several innovations, including lightweight optics, a  deployable sunshield, and a folding segmented mirror. Although the  program began risk reduction activities early to develop and mature some  technologies, such as the lightweight segmented folding mirror, the  program is challenged with maturing some of its other critical  technologies. For example, the sunshield, which consists of five layers of  membranes, must be folded for launch but then unfurled to its operational  configuration—with enough tension to prevent wrinkle patterns that could  interfere with the telescope’s mirrors, but not so much tension to cause  tears in the fabric. The sunshield must also be aligned with the rest of the  observatory so that only the top layer of the sunshield is visible to the  primary mirror and a correct angle between the observatory and the sun  and other heat-radiating bodies is maintained to enable the telescope and  science instruments to preserve the very cold temperature—about   40 degrees Kelvin—critical for achieving the JWST’s mission. In addition,  using passive cooling devices, such as heat switches, to allow specific  areas of the telescope to cool down, represent additional challenges since  these items will be used in new configurations. NASA also recently  substituted the cryo-cooler used for the mid-infrared instrument for a  lower technology component to save mass. According to JWST officials,  the program recently awarded the development contract for the cryo- cooler. In addition, the micro shutter array, which will allow the JWST to  program specific patterns of the electromagnetic spectrum for viewing, is  a new technology being developed by the Goddard Space Flight Center  and is still at a relatively low level of maturity. JWST officials acknowledge  that they are concerned about maturing the cryo-cooler and the micro  shutter array.
In addition, the program also faces design challenges related to the launch  vehicle and the observatory’s stability. For example, program officials told  us that they may need to request a waiver because the telescope will not fit  within the criteria limits of the launch vehicle’s envelop without making  design modifications. Furthermore, due to the late selection of the launch  vehicle, the project office and prime contractor are just beginning to  discuss interfaces, transportation at the launch site, and the additional  space issue with Ariane 5 officials. Also, the project faces the unresolved  problem of finding the best way to keep the observatory stable. The large  sunshield, observatory attitude changes, and other effects conspire to  produce unbalanced torques, which can make the observatory unstable.  The project continues to look at ways to resolve this problem, including  thrusters to rebalance the observatory, but project officials say this will  continue to be a challenge.
Another overriding concern is NASA’s inability to test the entire  observatory in its operational environment, since there is no test facility in  the United States large enough to perform this test. The plan is to  incrementally test components and subsystems on the ground in  laboratories simulating the observatory’s operational environment and to  make extensive use of modeling and simulation. According to the  memorandum summarizing the January 2006 System Definition Review, a  key concern is that the JWST is pushing the limits of ground test facilities  and cannot be tested at the observatory level; therefore, requiring  complicated integration and testing with a series of subsystem tests and  analyses. In its April 2006 assessment of the JWST program, the  Independent Review Team reported that there are several exceptions to  the “test as you fly” guideline and that mitigation strategies need to be  developed before the end of the preliminary design phase.
section_title: Containing Further Cost Growth and Schedule Slippage
paragraphs: In March 2005, the JWST program recognized that its cost had grown by  about $1 billion, increasing the JWST’s life-cycle cost estimate from   $3.5 billion to $4.5 billion. About half of the cost growth was due to  schedule slippage—a 1-year schedule slip because of a delay in the  decision to use an ESA-supplied Ariane 5 launch vehicle and an additional  10-month slip caused by budget profile limitations in fiscal years 2006 and  2007. More than a third of the cost increase was caused by requirements  and other changes. An increase in the program’s contingency funding  accounted for the remainder—about 12 percent—of the growth.
Despite an increase in the program’s contingency funding, the  Independent Review Team found that the contingency funding is still  inadequate. In its April 2006 assessment of the JWST program’s re- baselining, the Independent Review Team expressed concern over the  program’s contingency funding, stating that it is too low and phased in too  late. According to the team, the program’s contingency from 2006 through  2010 of only $29 million, or about 1.5 percent, after “liens” and “threats” is  inadequate. The team also stated that a 25 percent to 30 percent total  contingency is appropriate for a program of this complexity. The  program’s total contingency is only about 19 percent. The team warned  that because of the inadequate contingency, the program’s ability to  resolve issues, address program risk areas, and accommodate unknown  problems is very limited. Therefore, the team concluded that from a  budget perspective, the re-baselined program is not viable for a 2013  launch. The team recommended that before the Non-Advocate Review  (NAR) leading to program start, steps should be taken by the Science  Mission Directorate to assure that the JWST program contains an adequate  time-phased funding contingency to secure a stable launch date.
The JWST program remains at risk of incurring additional cost growth and  schedule slippage because of the technical challenges that must be  resolved—immature technologies, design challenges, and testing  restrictions. Our best practices work indicates that immature technology  increases the risk of cost increases and schedule slips. Unresolved  technology challenges can cascade through a product development cycle  often resulting in an unstable design that will require more testing and  thus more time and money to fix the problems. Subsequently, it will be  difficult to prepare a reliable cost estimate until these challenges are  resolved.
section_title: Knowledge-Based Approach Key to Overcoming Challenges
paragraphs: Our past work on the best practices of product developers in government  and industry has found that the use of a knowledge-based approach is a  key factor in successfully addressing challenges such as those faced by the  JWST program. Over the last several years, we have undertaken a body of  work on how leading developers in industry and government use a  knowledge-based approach to deliver high quality products on time and  within budget. A knowledge-based approach to product development  efforts enables developers to be reasonably certain that, at critical  junctures or “knowledge points” in the acquisition life cycle, their products  are more likely to meet established cost, schedule, and performance  baselines and therefore provides them with information needed to make  sound investment decisions. The marker for the first juncture—knowledge  point 1 (KP1)—occurs just prior to program start. At KP1, the customer’s  requirements match the product developer’s resources in terms of  knowledge, time, and money. At KP 2, the product design is stable, and  production processes are mature at KP 3. Product development efforts  that have not followed a knowledge-based approach can frequently be  characterized by poor cost, schedule, and performance outcomes.
We recently reported that NASA’s revised acquisition policy for developing  flight systems and ground support projects incorporates some aspects of  the best practices used by successful developers. For example, NASA  policy requires projects to conduct a major decision review—NAR— before moving from formulation to implementation. Further, before  moving from formulation to implementation, projects must validate  requirements and develop realistic cost and schedule estimates, human  capital plans, a preliminary design, and a technology plan—all key  elements for matching needs to resources before commitment to a major  investment is made at project start. Figure 2 compares NASA’s life cycle  with a knowledge-based acquisition life cycle.
NAR (KP1)
subsections: section_title: Knowledge Based Approach
subsections: section_title: Management Decision Reviews
paragraphs: While the policy incorporates elements of a knowledge-based approach,  we also reported that NASA’s acquisition policies lack the necessary  requirements to ensure that programs proceed and are funded only after  an adequate level of knowledge at key junctures. For example, NASA  policy does not require that programs demonstrate technologies at high  levels of maturity at program start. Further, although NASA policy does  require project managers to establish a continuum of technical and  management reviews, the policy does not specify what these reviews  should be nor does it require major decision reviews at other key points in  a product’s development. These best practices could be used to further  reduce program risks.
In order to close the gaps between NASA’s current acquisition  environment and best practices on knowledge-based acquisition, we  recommended that NASA take steps to ensure that NASA projects follow a  knowledge-based approach for product development. Specifically, we  recommended that NASA (1) in drafting its systems engineering policy,  incorporate requirements for flight systems and ground support projects  to capture specific product knowledge by key junctures in project  development and use demonstration of this knowledge as exit criteria for  decision making at key milestones and (2) revise NASA Procedural  Requirements 7120.5C to institute additional major decision reviews  following the NAR for flight systems and ground support projects, which  result in recommendations to the appropriate decision authority at key  milestones. NASA concurred with our recommendations and agreed to  revise its policies.
One of the resources needed at program start is mature technology. Our  best practices work has shown that technology readiness levels (TRL)—  a concept developed by NASA—can be used to gauge the maturity of  individual technologies. Specifically, TRL 6—demonstrating a technology  as a fully integrated prototype in a realistic environment—is the level of  maturity needed to minimize risks for space systems entering product  development. To achieve TRL 6, technology maturity must be  demonstrated in a relevant environment using a prototype or model.   (See app. II for a detailed description and definition of TRLs and test  environments.)
A knowledge-based approach also involves the use of incremental markers  to ensure that the required knowledge has been attained at each critical  juncture. For example, exit criteria at KP1 should include demonstrated  maturity of critical technologies, completed trade-offs and finalized  requirements, and initial cost and schedule estimates using results from  the preliminary design review. The approach ensures that managers will  (1) conduct activities to capture relevant product development knowledge,  (2) provide evidence that knowledge was captured, and (3) hold decision  reviews to determine that appropriate knowledge was captured to allow a  move to the next phase. If the knowledge attained at each juncture does  not justify the initial investment, the project should not go forward and  additional resources should not be committed.
section_title: Risks Not Fully Addressed by Recently Revised Acquisition Strategy
paragraphs: Prior to the program’s recent acquisition strategy revision, program  officials were not following NASA acquisition policy and were set to  commit to the program and start implementation with immature  technologies, according to best practices, and without a preliminary  design. For instance, the schedule called for convening the NAR before the  end of preliminary design. NASA policy indicates that the NAR and  Preliminary Design Review (PDR) should be aligned. Even at the pre- NAR in July 2003, the plan had been to have the NAR before the PDR,  although the two reviews were closer together than the more recent plan.
During our review, we discussed these shortfalls with NASA officials. To  their credit, they revised their acquisition strategy to conform to NASA  policy. Currently, the mission NAR—upon which the program start  decision will be based—will be aligned with the mission PDR (scheduled  for March 2008). We believe this is a positive step, since it will ensure that  a preliminary design—a key element for matching needs to resources—is  established before program start. The revised strategy also splits the NAR  into two parts—a technical NAR and a mission NAR. The purpose of the  technical NAR (scheduled for January 2007) will be to determine whether  the project has successfully retired its invention risk, i.e., critical  technologies have achieved TRL 6, according to a NASA official.  Technology issues will not be revisited after the technical NAR unless  problems arise. However, it is unclear if the critical technologies will be  demonstrated to a level of fidelity required by best practices at the  technical NAR. Furthermore, the strategy does not fully incorporate a  knowledge-based approach that could address the program’s risks by  ensuring—through the use of exit criteria—that resources match  requirements in terms of knowledge, time, and money before program  start. For example:    Under a knowledge-based approach, adequate testing is required to  demonstrate that key technologies are mature—at TRL 6—prior to  program start. This is particularly important for the JWST, given the  program’s challenges with testing restrictions and the fact that the  observatory cannot be serviced in space. In some cases, such as the  sunshield, backup technologies do not exist, thus increasing the  importance of adequately maturing and testing critical technologies. If  key components—like the sunshield—fail, then the entire observatory  will be lost. This requires greater fidelity in the testing, even as early as  demonstrating the maturity of key technologies prior to program start.
To achieve TRL 6 (the maturity level required by best practices for  program start), technology maturity must be demonstrated as a  representative model or prototype—which is very close to the actual  system in form, fit, and function—in a relevant environment. However,  there is risk that the current JWST technology development plan will  not result in the appropriate demonstration of technology maturity. For  example, the half-scale thermal vacuum test of the entire observatory  at Johnson Space Center is currently planned for September 2008, and  so the knowledge gained regarding the maturity of the sunshield’s  thermal and dynamic performance is pushed out 6 months beyond the  PDR/NAR/program start date of March 2008. When JWST program  officials briefed us in August 2005, the TRL levels for thermal and  dynamic performance of the sunshield were both assessed to be at TRL  4, and the plan to get to TRL 6 was to test these subsystems during this  half-scale thermal vacuum test. However, in fall 2005 program officials  reviewed the technology development plan and concluded that only the  materials for the sunshield’s membrane are technology development  items, while other items affecting the configuration and deployment of  the sunshield—such as thermal and dynamic performance—are  considered engineering challenges. JWST officials stated that earlier  testing of sample materials demonstrated the sunshield’s thermal  performance and a demonstration using a 1/10th scale model  demonstrated dynamic performance and satisfied TRL 6 requirements.  However, we have found in our best practices work that demonstrating  a technology to a TRL 6 typically involves demonstrating that a  prototype—close to the form, fit, and functionality intended for the  product—has been demonstrated in an environment that closely  represents the anticipated operational environment. In our past review  of development programs, we have found that if this level of maturity is  not demonstrated before a product development effort is launched, a  program increases the likelihood of cost growth and schedule delays as  it tries to close the knowledge gap between the technologies’ maturity  level and the product’s design requirements.
The JWST program’s inadequate contingency runs contrary to another  premise of a knowledge-based approach—having sufficient resources  in terms of funding available to ensure a program’s success. As  discussed in an earlier section, the Independent Review Team stated  that the program’s contingency from 2006 through 2010 of only about  1.5 percent after “liens” and “threats” is inadequate. The team warned  that, because of the inadequate contingency, the program’s ability to  resolve issues, address program risk areas, and accommodate  unknown problems is very limited. The team concluded that, from a  budget perspective, the re-baselined program is not viable for a 2013  launch.
section_title: Knowledge-Based Approach Would Allow the JWST Program to Better Inform NASA’s Decision- Making Process
paragraphs: A good basis for making informed investment decisions is essential in the  fiscally constrained environment that now exists across the federal  government. Our nation faces large, growing, and structural long-term  fiscal imbalances. Given the severity of those fiscal challenges and the  wide range of federal programs, hard choices need to be considered  across the government, and NASA is no exception. NASA must compete  with other departments and agencies for part of a constricted  discretionary spending budget.
In the near future, NASA will need to determine the resources necessary to  develop the systems and supporting technologies to achieve the  President’s Vision for Space Exploration—while simultaneously financing  its other priority programs—and structure its investment strategy  accordingly. Initial implementation of the Vision as explained in NASA’s  Exploration Systems Architecture Study calls for completing the  International Space Station, developing a new crew exploration vehicle,  and returning to the moon no later than 2020. NASA estimates that it will  cost approximately $104 billion over the next 13 years to accomplish these  initial goals. These priorities, along with NASA’s other missions, will be  competing within NASA for funding. It will likely be difficult for decision  makers to agree on which projects to invest in and which projects, if any,  to terminate. The NASA Administrator has acknowledged that NASA faces  difficult choices about its missions in the future—for example, between  human space flight, science, and aeronautics missions.
In the President’s fiscal year 2007 budget request for NASA, the JWST has  the largest budget allocation of all programs in the Science Mission  Directorate’s Astrophysics Division for the 5-year budget horizon from  fiscal year 2007 through fiscal year 2011—nearly $2 billion of the division’s  $6.9 billion total budget, or about 29 percent. An inadequately informed  decision to commit to the estimated $4.5 billion total funding for the JWST  would significantly impact NASA’s science portfolio, since funding given  to the JWST will not available for other programs. Early in the planning for  how to handle the JWST program’s cost growth, NASA officials recognized  the impact that the JWST’s cost growth could have on other programs. In a  July 2005 briefing to the Agency Program Management Council soon after  the cost growth was identified, NASA officials stated that “something must  give if JWST stays in the portfolio.” The choices discussed were   (1) relaxing requirements or (2) adding budget and schedule, which would  mean that other missions would be deferred or deleted from the portfolio.
In addition, committing to the JWST program obligates the government  contractually, since it allows the prime contractor to begin  implementation tasks on the very long prime contract extending from  October 2002 through launch—currently planned for June 2013—plus one  year. The contract states that until the project achieves the  implementation milestone, contract spending is limited to formulation  activities, except for long-lead items and other activities approved in  writing. After the implementation milestone is achieved at program start,  the contracting officer will notify the contractor by letter to proceed to  implementation. According to the contracting officer, the assumption is  that this is the go-ahead for the whole program.
To make well-informed decisions, NASA needs the knowledge to assess  the value of its programs—like the JWST program—in relationship to each  other. In May 2004, we reported that, of 27 NASA programs we examined,  17 had cost increases averaging about 31percent. One of the programs in  our sample was another infrared telescope program—the Spitzer Space  Telescope—and it was plagued by schedule slippages caused by delays in  the delivery of components, flight software, the mission operation system,  and launch delays, all contributing to a 29.3 percent increase in program  costs. In general, we found the programs in the sample lacked sufficient  knowledge needed to make informed acquisition decisions. Insufficient  knowledge to make informed investment decisions can further complicate  the already-difficult choices that NASA faces. Conversely, sufficient  knowledge at key junctures can facilitate well-informed investment  decisions and protect the government from incurring contractual liabilities  before it is appropriate. A knowledge-based approach ensures that  comprehensive and comparable programmatic data are obtained.
section_title: Conclusions
paragraphs: Within the JWST program, NASA officials have accomplished a great deal,  such as the development of the large, segmented mirror that is a leap  ahead in technology. Moreover, the program has support from the larger  scientific community. To enhance the program’s chances for success,  program officials have chosen a path forward which follows NASA’s  policies for ensuring readiness to proceed into implementation/product  development. However, the JWST program’s revised strategy does not  fully address the risks associated with the many challenges that the  program still faces—including maturing technology, mitigating testing  restrictions, and ensuring that adequate funding is available for  contingencies. This puts the program at risk of further cost growth and  schedule slippage. The program needs to have sufficient knowledge at key  junctures to successfully address its challenges and use incremental  markers to make certain that resources in terms of knowledge, time,  workforce, and money match the requirements. Given the severity of the  fiscal challenges our nation faces and the wide range of competing federal  programs, hard choices need to be considered across the government, and  NASA is no exception. Using a knowledge-based approach for NASA’s new  development programs such as the JWST could help the agency make the  difficult choices about how to allocate its limited budget resources among  competing priorities by utilizing common and consistent criteria in  program evaluations.
section_title: Recommendations for Executive Action
paragraphs: To increase the JWST program’s chances of successful product  development, we recommend that the NASA Administrator take the  following actions:
Direct the JWST program to fully apply a knowledge-based acquisition  approach—to include incremental markers—that will not only ensure  that adequate knowledge is attained at key decision points, but also  hold the program accountable. These markers should include, but not  be limited to    schedules that demonstrate the maturity of all critical  technologies prior to program start;    criteria to ensure the validity of test articles;    criteria to demonstrate that mature component designs being  used in new configurations meet form, fit, and function  standards; and    criteria to ensure that sufficient contingency funding can be  provided and phased appropriately.
Instruct the JWST program to continue to adhere to NASA acquisition  policy and base the program’s go/no-go review (NAR) decision not  only on adherence to that policy, but also on (1) the program’s ability  to demonstrate whether it is meeting the knowledge markers outlined  earlier and (2) whether adequate funds are available to execute the  program.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, NASA concurred with our  two recommendations and outlined actions that the agency plans to take  to implement such recommendations. NASA said that it endorses the  knowledge-based approach recommended and that it believes the current  JWST program plan is consistent with that approach.  NASA’s recognition  of the value of obtaining knowledge prior to moving to subsequent  acquisition phases and acknowledgment that it plans to use exit criteria as  knowledge markers for other JWST mission-level reviews are welcome  steps toward establishing an agency-wide risk reduction culture. Now, it  will be critical for NASA decision makers to enforce adherence to the  discipline of the knowledge-based approach and ensure that critical  product knowledge is indeed demonstrated before allowing the JWST  program to proceed. In the years ahead, NASA decision makers will likely  face pressures to grant waivers for going forward with immature  technologies, allow programs to be restructured, and thus marginalize  accountability. For a program such as the JWST, whose investment is  already substantial and successful outcome eagerly anticipated by the  science community, adherence to such knowledge-based principles will  need to be strictly enforced. As identified in this report, NASA would be  well served by applying its own technology readiness standards (reprinted  in appendix II) as part of its exit criteria, and demonstrating that critical  technologies are at the TRL 6 level prior to program start using a  representative model or prototype—which is very close to the actual  system in form, fit, and function—in a relevant environment. Emphasis by  decision makers on the application of “form, fit, and function standards”  and “validity of test articles” as exit criteria for the JWST program start  and entry into Phase C will help address our concern that the current  JWST technology development plan may not result in the appropriate  demonstration of technology maturity prior to program start. NASA’s  comments are reprinted in appendix III.
We are sending copies of this report to interested congressional  committees and to the NASA Administrator. We will make copies available  to others upon request. In addition, the report will be available at no  charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (202) 512-4841 or lia@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. Key contributors to this report are acknowledged in  appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: To assess the extent to which the JWST acquisition strategy follows NASA  policy and GAO best practices for ensuring readiness to proceed into  implementation, we reviewed NASA policy on program management and  compared the JWST project office’s management approach to NASA  policy. Additionally, we analyzed the JWST acquisition strategy and  benchmarked it to best practices. We interviewed NASA and contractor  officials to clarify our understanding of the JWST management approach  and technology development plan in relation to NASA policy and  guidelines and best practices. To deepen our understanding of JWST  technical issues, we attended the 3-day Sunshield Subsystem Concept  Design Review as well as the 4-day JWST System Definition Review.
To evaluate the impact of the JWST acquisition strategy on NASA’s ability  to assess the program and make informed investment decisions in the  context of its other priorities, we analyzed available JWST cost and  schedule data and conducted interviews with program officials to clarify  our understanding of the information. Furthermore, we requested and  reviewed documentary support breaking out the components of the cost  increases and schedule slippage. We also interviewed program officials to  clarify our understanding of the potential impact that investment in the  JWST will have on other NASA programs. In addition, we reviewed  statements of the NASA Administrator, budget documents, GAO’s High- Risk Series, and GAO’s 21st Century Challenges to better evaluate the  JWST’s significance in the larger NASA and federal government context.
To accomplish our work, we visited NASA Headquarters, Washington,  D.C.; Goddard Space Flight Center, Greenbelt, Maryland; Marshall Space  Flight Center, Huntsville, Alabama; Northrop Grumman Space Technology,  Redondo Beach, California; and Ball Aerospace and Technologies  Corporation, Boulder, Colorado.
We performed our review from August 2005 through May 2006 in  accordance with generally accepted government auditing standards.
section_title: Appendix II: Technology Readiness Levels
subsections: section_title: Lowest level of technology readiness. Scientific research begins to be translated into applied research and development. Examples might include paper studies of a technology’s basic properties.
paragraphs: None. (Paper studies and  analysis.)
Invention begins. Once basic principles  are observed, practical applications can  be invented. The application is  speculative and there is no proof or  detailed analysis to support the  assumption. Examples are still limited to  paper studies.
None. (Paper studies and  analysis.)
Active research and development is  initiated. This includes analytical studies  and laboratory studies to physically  validate analytical predictions of  separate elements of the technology.  Examples include components that are  not yet integrated or representative.
Analytical studies and  demonstration of nonscale  individual components  (pieces of subsystem).
Basic technological components are  integrated to establish that the pieces  will work together. This is relatively “low  fidelity” compared to the eventual  system. Examples include integration of  “ad hoc” hardware in a laboratory.
Low fidelity breadboard.  Integration of nonscale  components to show pieces  will work together. Not fully  functional or form or fit but  representative of technically  feasible approach suitable for  flight articles.
Fidelity of breadboard technology  increases significantly. The basic  technological components are  integrated with reasonably realistic  supporting elements so that the  technology can be tested in a simulated  environment. Examples include “high  fidelity” laboratory Integration of  components.
High fidelity breadboard.  Functionally equivalent but  not necessarily form and/or fit  (size weight, materials, etc.).  Should be approaching  appropriate scale. May  include integration of several  components with reasonably  realistic support  elements/subsystems to  demonstrate functionality.
Lab demonstrating  functionality but not form  and fit. May include flight  demonstrating  breadboard in surrogate  aircraft. Technology  ready for detailed design  studies.
Representative model or prototype  system, which is well beyond the  breadboard tested for TRL 5, is tested in  a relevant environment. Represents a  major step up in a technology’s  demonstrated readiness. Examples  include testing a prototype in a high  fidelity laboratory environment or in  simulated operational environment.
Prototype—Should be very  close to form, fit and function.  Probably includes the  integration of many new  components and realistic  supporting  elements/subsystems if  needed to demonstrate full  functionality of the  subsystem.
High-fidelity lab  demonstration or  limited/restricted flight  demonstration for a  relevant environment.  Integration of technology  is well defined.
Prototype near or at planned operational  system. Represents a major step up  from TRL 6, requiring the demonstration  of an actual system prototype in an  operational environment, such as in an  aircraft, vehicle or space. Examples  include testing the prototype in a test  bed aircraft.
Prototype. Should be form, fit  and function integrated with  other key supporting  elements/subsystems to  demonstrate full functionality  of subsystem.
Flight demonstration in  representative  operational environment  such as flying test bed or  demonstrator aircraft.  Technology is well  substantiated with test  data.
Technology has been proven to work in  its final form and under expected  conditions. In almost all cases, this TRL  represents the end of true system  development. Examples include  developmental test and evaluation of  the system in its intended weapon  system to determine if it meets design  specifications.
Developmental test and  evaluation in the actual  system application.
Actual application of the technology in  its final form and under mission  conditions, such as those encountered  in operational test and evaluation. In  almost all cases, this is the end of the  last “bug fixing” aspects of true system  development. Examples include using  the system under operational mission  conditions.
Operational test and  evaluation in operational  mission conditions.
section_title: Appendix III: Comments from the National Aeronautics and Space Administration
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Jim Morrison, Assistant  Director; Greg Campbell; Keith Rhodes; Sylvia Schatz; Erin Schoening;   Hai Tran; and Ruthie Williamson made key contributions to this report.
section_title: Related GAO Products
paragraphs: NASA: Implementing a Knowledge-Based Acquisition Framework Could  Lead to Better Investment Decisions and Project Outcomes. GAO-06-218.  Washington, D.C.: December 21, 2005.
NASA’s Space Vision: Business Case for Prometheus 1 Needed to Ensure  Requirements Match Available Resources. GAO-05-242. Washington, D.C.:  February 28, 2005.
subsections: section_title: Space Reports
paragraphs: Space Acquisitions: Stronger Development Practices and Investment  Planning Need to Address Continuing Problems. GAO-05-891T.  Washington, D.C.: July 12, 2005.
Defense Acquisitions: Incentives and Pressures That Drive Problems  Affecting Satellite and Related Acquisitions. GAO-05-570R. Washington,  D.C.: June 23, 2005.
Defense Acquisitions: Space-Based Radar Effort Needs Additional  Knowledge before Starting Development. GAO-04-759. Washington, D.C.:  July 23, 2004.
Defense Acquisitions: Risks Posed by DOD’s New Space Systems  Acquisition Policy. GAO-04-379R. Washington, D.C.: January 29, 2004.
Space Acquisitions: Committing Prematurely to the Transformational  Satellite Program Elevates Risks for Poor Cost, Schedule, and  Performance Outcomes. GAO-04-71R. Washington, D.C.: December 4,  2003.
Defense Acquisitions: Improvements Needed in Space Systems  Acquisition Policy to Optimize Growing Investment in Space.   GAO-04-253T. Washington, D.C.: November 18, 2003.
Defense Acquisitions: Despite Restructuring, SBIRS High Program  Remains at Risk of Cost and Schedule Overruns. GAO-04-48. Washington,  D.C.: October 31, 2003.
Defense Acquisitions: Improvements Needed in Space Systems  Acquisition Management Policy. GAO-03-1073. Washington, D.C.:  September 15, 2003.
Military Space Operations: Common Problems and Their Effects on  Satellite and Related Acquisitions. GAO-03-825R. Washington, D.C.: June  2, 2003.
Military Space Operations: Planning, Funding, and Acquisition  Challenges Facing Efforts to Strengthen Space Control. GAO-02-738.  Washington, D.C.: September 23, 2002.
Polar-Orbiting Environmental Satellites: Status, Plans, and Future Data  Management Challenges. GAO-02-684T. Washington, D.C.: July 24, 2002.
Defense Acquisitions: Space-Based Infrared System-Low at Risk of  Missing Initial Deployment Date. GAO-01-6. Washington, D.C.: February  28, 2001.
section_title: Best Practices Reports
paragraphs: Defense Acquisitions: Assessments of Selected Major Weapon Programs.  GAO-05-301. Washington, D.C.: March 31, 2005.
Defense Acquisitions: Stronger Management Practices Are Needed to  Improve DOD’s Software-Intensive Weapon Acquisitions. GAO-04-393.  Washington, D.C.: March 1, 2004.
Defense Acquisitions: Assessments of Selected Major Weapon Programs.  GAO-04-248. Washington, D.C.: March 31, 2004.
Defense Acquisitions: DOD’s Revised Policy Emphasizes Best Practices,  but More Controls Are Needed. GAO-04-53. Washington, D.C.: November  10, 2003.
Defense Acquisitions: Assessments of Selected Major Weapon Programs.  GAO-03-476. Washington, D.C.: May 15, 2003.
Best Practices: Setting Requirements Differently Could Reduce Weapon  Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February  11, 2003.
Best Practices: Capturing Design and Manufacturing Knowledge Early  Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15,  2002.
Defense Acquisitions: DOD Faces Challenges in Implementing Best  Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.
Best Practices: Better Matching of Needs and Resources Will Lead to  Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8,  2001.
Best Practices: A More Constructive Test Approach Is Key to Better  Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31,  2000.
Defense Acquisition: Employing Best Practices Can Shape Better  Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April  26, 2000.
Best Practices: DOD Training Can Do More to Help Weapon System  Program Implement Best Practices. GAO/NSIAD-99-206. Washington,  D.C.: August 16, 1999.
Best Practices: Better Management of Technology Development Can  Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington,  D.C.: July 30, 1999.
Defense Acquisitions: Best Commercial Practices Can Improve Program  Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.
Defense Acquisition: Improved Program Outcomes Are Possible.   GAO/T-NSIAD-98-123. Washington, D.C.: March 18, 1998.
Best Practices: Successful Application to Weapon Acquisition Requires  Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.:  February 24, 1998.
Major Acquisitions: Significant Changes Underway in DOD’s Earned  Value Management Process. GAO/NSIAD-97-108. Washington, D.C.: May 5,  1997.
Best Practices: Commercial Quality Assurance Practices Offer  Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26,  1996.