James Webb Space Telescope: Actions Needed to Improve Cost Estimate and Oversight of Test and Integration
Released: 2012-12-03T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: JWST is one of NASA's most expensive and technologically advanced science projects, intended to advance understanding of the origin of the universe. In 2011, JWST was rebaselined with a life cycle cost estimate of $8.8 billion and a launch readiness date in October 2018--almost nine times the cost and more than a decade later than originally projected in 1999. Concern about the magnitude of JWST's cost increase and schedule delay and their effects on NASA's progress on other high-priority missions led conferees for the Consolidated and Further Continuing Appropriations Act, 2012, to direct GAO to report on the project. Specifically, GAO assessed (1) the extent to which NASA's revised cost and schedule estimates are reliable based on best practices, (2) the major risks and technological challenges JWST faces, and (3) the extent to which NASA has improved oversight of JWST. To do this, GAO compared NASA's revised cost and schedule estimates with best practice criteria, reviewed relevant contractor and NASA documents, and interviewed project and contractor officials.

section_title: What GAO Found
paragraphs: The National Aeronautics and Space Administration (NASA) has provided significantly more time and money to the James Webb Space Telescope (JWST) than previously planned and expressed high confidence in the project's new baselines. Its current cost estimate reflects some features of best practices for developing reliable and credible estimates. For example, the estimate substantially meets one of four cost characteristics--comprehensive--that GAO looks for in a reliable cost estimate, in part because all life cycle costs were included. The estimate, however, only partially met the other three characteristics--well documented, accurate, and credible--which detracts from its reliability. For example, the estimate's accuracy, and therefore the confidence level assigned to the estimate, was lessened by the summary schedule used for the joint cost and schedule risk analysis because it did not provide enough detail to determine how risks were applied to critical project activities. The estimate's credibility was also lessened because officials did not perform a sensitivity analysis that would have identified key drivers of costs, such as workforce size. Program officials believe that it would have been difficult to fully address all best practice characteristics. GAO believes there is time to improve the estimate and enhance the prospects for delivering the project according to plan.

Project officials report that the JWST schedule has 14 months of reserve, which meets Goddard guidance for schedule reserve; however, only 7 of the 14 months are likely to be available for the last three of JWST's five complex integration and test efforts. GAO's prior work shows that the integration and test phases are where problems are commonly found and schedules tend to slip. Given that JWST has a challenging integration and test schedule, this could particularly be likely. The project has made some significant progress in the past year, notably successfully completing development of the 18 primary mirror segments--considered JWST's top technical risk. Nevertheless, ongoing challenges are indicative of the kinds of issues that can require significant effort to address. For example, instrument challenges have delayed the first integration and test effort. In addition, key long-term risks on subsystems with a significant amount of work remaining will not be retired until 2016. Currently, NASA's plan for project oversight calls for one independent mission-level system integration review about 13 months before launch. While this is consistent with what NASA requires for its projects, this approach may not be sufficient for a project as complex as JWST.

JWST has taken several steps to improve communications and oversight of the project and its contractors--such as taking over responsibility for mission systems engineering from the prime contractor; instituting meetings that include various levels of NASA, contractor, and subcontractor management; and implementing a new risk management system to allow for better tracking of risks. The enhancements to the oversight of the project are steps in the right direction, but it will take time to assess their effectiveness and ensure that the efforts are sustained by the project in the future. Reductions in travel budgets, however, could require the project to adjust the oversight approach that was adopted as a result of the replan. Additional reductions in travel budgets are anticipated in future years, but officials do not have a plan to address such reductions and their potential impact on continuing the current oversight approach.

section_title: What GAO Recommends
paragraphs: GAO recommends NASA take six actions including, among others, to take steps to improve its cost estimate; to conduct an additional, earlier independent review of test and integration activities; and to develop a long-term oversight plan that anticipates planned travel budget reductions.

section_title: Letter
paragraphs: The James Webb Space Telescope (JWST) is one of the National  Aeronautics and Space Administration’s (NASA) most expensive and  technologically advanced science projects. It is intended to revolutionize  our understanding of star and planet formation and advance the search  for the origins of our universe. The project has been hampered by  ineffective management, unrealistic estimates of costs, and insufficient  resources, and is projected to cost significantly more and to take much  longer to complete than expected when it was authorized to proceed in  1999. In 2011, JWST was rebaselined with a lifecycle cost estimate of  $8.8 billion and a launch date in October 2018, which is almost nine times  the cost and more than a decade later than originally projected. The  magnitude of JWST’s cost increase and schedule delay has and will  continue to have a major effect on NASA’s ability to implement or begin  high-priority science missions for years to come.

NASA has expressed high confidence in being able to execute the project  within its new cost and schedule baselines, and officials report that the  project is being executed to its revised estimates in the year since the  rebaseline. The project, however, is less than halfway through its  development with known and unknown technical risks and challenges  remaining to be addressed. The on-time and on-budget delivery of JWST  is a high congressional priority. In November 2011, the Joint Explanatory  Statement of the Committee of Conference for the Consolidated and  Further Continuing Appropriations Act, 2012, recommended that JWST’s  formulation and development costs, not including the cost to operate the  mission after launch, be no more than $8 billion. The conferees also  directed GAO to assess the program annually and to report to the  Committees on Appropriations on key issues relating to program and risk  management, achievement of cost and schedule goals, and program  technical status. This report responds to that direction. Specifically, we  assessed (1) the extent to which NASA’s revised cost and schedule  estimates are reliable based on GAO best practices, (2) the major risks  and technological challenges the JWST project faces, and (3) the extent  to which NASA has improved the oversight of the JWST project.

Our approach included an examination of the revised cost and schedule  estimates NASA developed during the replanning process using GAO’s  best practice criteria. This included review of documentation of the inputs  that supported the detailed cost and schedule estimates used by  decision-makers to rebaseline the program and interviews with project  officials involved in developing the estimates. We also reviewed the  proposal from the JWST prime contractor used as input to the replan and  interviewed contractor officials. We did not assess the most current JWST  integrated master schedule because of ongoing negotiations for contract  modifications associated with the replan. To identify the major short and  long term risks and technological challenges facing the project, we  interviewed project officials for the major subsystems as well as  contractor officials. We also reviewed the project’s risk database, monthly  status reviews, and other documentation provided by the project officials.  To understand the steps NASA implemented to improve the oversight  being performed on the project, we interviewed project and contractor  officials and reviewed documentation from independent reviews prior to  the replan and actions taken by NASA in response. We also interviewed  the Director of NASA’s Office of Evaluation concerning a recent  independent review of the JWST project performed by his office. We took  appropriate steps to address data reliability, such as clarifying data  discrepancies and corroborating NASA-generated data with other sources  where applicable. NASA provided comments and technical clarifications  on a draft of this report, which were incorporated as appropriate.  Appendix I contains detailed information on our scope and methodology.

We conducted this performance audit from February 2012 to December  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Initially referred to as the “Next Generation Space Telescope,” JWST is a  large deployable, infrared-optimized space telescope intended to be the  successor to the aging Hubble Space Telescope. JWST is designed to be  a 5-year mission to find the first stars and trace the evolution of galaxies  from their beginning to their current formation, and is intended to operate  in an orbit approximately 1.5 million kilometers—or 1 million miles—from  the Earth. In a 2001 decadal survey, the National Research Council rated  the JWST as the top-priority new initiative for astronomy and physics.  With its 6.5-meter primary mirror, JWST will be able to operate at 100  times the sensitivity of the Hubble Space Telescope. A tennis-court-sized  sunshield will protect the mirrors and instruments from the sun’s heat to  allow the JWST to look at very faint infrared sources. The Hubble Space  Telescope operates primarily in the visible and ultraviolet regions.
subsections: section_title: History of Cost Growth and Schedule Delays
paragraphs: JWST has experienced significant increases to project costs and  schedule delays. Prior to being approved for development, cost estimates  of the project ranged from $1 billion to $3.5 billion with expected launch  dates ranging from 2007 to 2011. In March 2005, NASA increased the  JWST’s life-cycle cost estimate to $4.5 billion and slipped the launch date  to 2013. We reported in 2006 that about half of the cost growth was due  to schedule slippage—a 1-year schedule slip because of a delay in the  decision to use a European Space Agency-supplied Ariane 5 launch  vehicle and an additional 10-month slip caused by budget profile  limitations in fiscal years 2006 and 2007. More than a third of the cost  increase was caused by requirements and other changes. An increase in  the program’s contingency funding accounted for the remainder—about  12 percent—of the growth. NASA Headquarters chartered an  Independent Review Team to evaluate the project that same year. In April  2006, the review team’s assessment confirmed that the program’s  technical content was complete and sound, but expressed concern over  the project’s contingency reserve funding—funding used to mitigate  issues that arise but which were previously unknown—reporting that it  was too low and phased in too late in the development life cycle. The  team reported that for a project as complex as the JWST, a 25 to 30  percent total contingency was appropriate. At that time, JWST’s total  contingency was about 19 percent. The team cautioned that this  contingency compromised the project’s ability to resolve issues, address  risk areas, and accommodate unknown problems. The team also  concluded that the 2013 launch date was not viable for the project based  on its anticipated budget. It recommended that before the project was  formally approved for development and baselined, NASA should take  steps to provide the JWST project with adequate time-phased reserve  funding to secure a stable launch date. Additional reserves were added  and the project was baselined in April 2009 with a life-cycle cost estimate  of $4.964 billion and a launch date in June 2014.

Shortly after JWST was approved for development and its cost and  schedule estimates were baselined, project costs continued to increase.  In 2010, Senator Barbara Mikulski, chair of the Senate Committee on  Appropriations, Subcommittee on Commerce, Justice, Science, and  Related Agencies, asked NASA to initiate another independent review in  response to the project’s cost increases and reports that the June 2014  launch date was in jeopardy. The Independent Comprehensive Review  Panel (ICRP) was commissioned by NASA and began its review in  August 2010. In October 2010, the ICRP issued its report and cited  several reasons for the project’s problems including management,  budgeting, oversight, governance and accountability, and communication  issues. The panel concluded JWST was executing well from a technical  standpoint, but that the baseline funding did not reflect the most probable  cost with adequate reserves in each year of project execution, resulting in  an unexecutable project. The review panel recommended that additional  resources be considered along with organizational and management  restructuring. Following this review, the JWST program underwent a  replan in 2011. In November 2011, the JWST project was reauthorized,  but not before it was recommended for termination by the House  Appropriations Committee. On the basis of the replan, NASA announced  that the project would be rebaselined at $8.835 billion—a 78 percent  increase to the project’s life-cycle cost from the confirmed baseline—and  would launch in October 2018—a delay of 52 months. The revised life- cycle cost estimate included 13 months of funded schedule reserve. In  the President’s Fiscal Year 2013 budget request, NASA reported a 66  percent joint cost and schedule confidence level associated with these  estimates. A joint cost and schedule confidence level (JCL) is the process  NASA uses to assign a percentage to the probable success of meeting  cost and schedule targets and is part of the project’s estimating process.

section_title: Current JWST Organization
paragraphs: The JWST project is divided into three major segments: the launch  segment, the ground segment, and the observatory segment. The launch  segment is primarily provided by the European Space Agency (ESA),  which is contributing the Ariane 5 launch vehicle and launch site  operations in French Guiana. The ground segment will be responsible for  collecting the data obtained by JWST in space and making it usable for  scientists and researchers. This includes the development of software  that will translate data into usable formats as well as operation of the  software once the telescope is in space. The Space Telescope Science  Institute, operated by the Association of Universities for Research in  Astronomy (AURA) on a contract awarded by NASA, which currently  performs science operations for the Hubble Space Telescope, is  developing the science and operations and flight operations center for  JWST and will conduct the first 6 months of flight and science operations.  The NASA contract with the Space Telescope Science Institute extends  through the first 6 months of JWST operations. A contract to manage the  long term operations of JWST is planned to be awarded approximately 2  years prior to launch. The observatory segment will be launched into  space and includes five major subsystems. These subsystems are being  developed through a mixture of NASA, contractor, and international  partner efforts. See figure 1.

JWST is a single project program reporting directly to the NASA  Associate Administrator for programmatic oversight and to the Associate  Administrator for the Science Mission Directorate for technical and  analysis support. Goddard Space Flight Center is the NASA center  responsible for the management of JWST. See figure 2 for the current  JWST organizational chart.

section_title: Revised JWST Cost Estimate Is Not Fully Consistent with Best Practices for Reliable and Credible Estimates and the Integrated Master Schedule Is Not Finalized
paragraphs: Our analysis of JWST’s revised cost estimate showed that it is not fully  consistent with best practices for developing reliable and credible  estimates, although project officials took some steps in line with best  practices in the development of the estimate. For example, as part of its  cost estimation process, the project conducted a joint cost and schedule  risk analysis, or joint cost and schedule confidence level (JCL), which  assigned a 66 percent confidence level to the estimate. In addition, we  found that the cost estimate included all life cycle costs for the project.  Although NASA’s methods for developing the JWST cost estimate reflect  some features of best practices, our review of the estimate showed that  based on best practice criteria, it did not fully meet the four characteristics  of a reliable estimate. See figure 3.

Specifically, the project’s estimate was found to substantially meet the  best practice criteria for being comprehensive, and the remaining three  characteristics of being well documented, accurate, and credible were  found to be only partially met. For example, the accuracy of the cost  estimate, and therefore the confidence level assigned to the estimate,  was lessened by the schedule used in the JCL analysis because it  prevented us from, among other things, identifying the activities that were  on the critical path—defined as time associated with activities that drive  the overall schedule. The credibility of the estimate was lessened  because project officials did not perform a sensitivity analysis that would  have identified key drivers of costs, such as workforce size. Although  NASA is not required to adhere to these best practices, our prior work has  shown that not following best practices for cost estimating can make the  cost estimate less reliable, putting projects at risk of experiencing cost  overruns, missed deadlines, and performance shortfalls. The best  practices stem from practices federal cost estimating organizations and  industry use to develop and maintain reliable cost estimates, including the  Department of Defense and NASA. According to program officials, it  would have been difficult, if not impossible, for the project to have met all  of the best practice criteria given the complexity of the project and that  some elements of the project are quite mature in their development.  Instead, the program manager stated that the project followed a tailored  process to develop the cost estimate that was appropriate for the project.  Furthermore, officials report the project is currently meeting a majority of  its milestones and executing as planned to the revised estimates for the  JWST.

A work breakdown structure reflects the requirements and what must be accomplished to  develop a program, and it provides a basis for identifying resources and tasks for  developing a program cost estimate. The work breakdown structure should be used to  define all program activities and tasks to ensure that the schedule encompasses the entire  work. two was not compatible. Finally, although the project outlined and  documented the ground rules and assumptions, we were unable to  determine whether risks associated with any assumptions were identified  and traced to specific elements.

Well documented: The JWST cost estimate only partially met the criteria  for being well documented because it did not include a step-by-step  description of how the estimate was developed, the raw data used to  develop the estimate, or the calculations and estimating methodology for  specific cost elements of the work breakdown structure. Without good  documentation, a cost analyst unfamiliar with the program will not be able  to replicate the estimate, because he or she will not understand the logic  behind it. Good documentation, for example, assists management and  oversight in assessing the credibility of the estimate, helps to keep a  history of reasons for cost changes and to record lessons learned,  defines the scope of the analysis, and answers questions about the  approach or data used to create the estimate. Project documentation,  however, does provide evidence that NASA management reviewed and  accepted the cost estimate because managers were briefed on the  technical aspects of the estimate and were provided an overview of the  joint cost and schedule risk analysis that was conducted.

Accurate: The JWST cost estimate only partially met the criteria for being  accurate because the projected costs of schedule reserve did not reflect  actual data, the summary schedule used to derive the JCL prevented us  from sufficiently understanding how risks were incorporated, and the  project did not provide evidence that it regularly updates the estimate or  plans to conduct another JCL. For example, using historical actual cost  data from Northrop Grumman, we estimated that 13 months of schedule  reserve is likely to be $204 million instead of NASA’s estimate of $121  million—a potential underestimation of 69 percent related to the schedule  reserve. Project officials, however, believe they have adequate reserves  available to offset any underestimation. In addition, the summary  schedule the project used as an input to the JCL, although deemed  acceptable by NASA, contained many long-duration activities, some with  1,000 days or more. Because of these long durations in the summary  schedule used for the JCL, the lack of detail prevented us from identifying  the activities that were on the critical path, as well as which risks were  applied to remaining activities. As a result, there is no way to ensure  that risks were appropriately assigned to activities in the schedule to  account for the impact of the risks during the JCL analysis.

Finally, it was unclear whether the cost estimate was regularly updated to  reflect material changes in actual costs and in the project itself, such as  when schedules or other assumptions change, due to a lack of detailed  documentation for the cost estimate. Project officials stated that in  keeping with NASA policy they do not plan, nor are they required, to  conduct another JCL analysis. GAO’s cost estimating best practices call  for estimates to be continually updated through the life of the project,  ideally every month as actual costs are reported in earned value  management reports, and that a risk analysis and risk simulation  exercise—like the JCL analysis—be conducted periodically through the  life of the program, as risks can materialize or change throughout the life  of a project. Unless properly updated on a regular basis, the cost estimate  cannot provide decision makers with accurate information to assess the  current status of the project. NASA officials state that the life-cycle cost  estimate is updated annually for the budgeting process, and that historical  records such as earned value data were used to develop the estimate.  They also stated that this information is updated in several different  documents being provided to management; however, we were unable to  determine how this information was used in updating the cost estimate on  a regular basis.

Credible: The JWST cost estimate only partially met the criteria for being  credible because project officials did not adequately test and verify the  reasonableness of the cost estimate and the schedule used in conducting  the JCL did not have a valid critical path and contained durations that  were too long to properly account for risks. For example, project officials  said they did not perform a sensitivity analysis for the cost estimate. A  sensitivity analysis identifies key elements that drive cost and permits  analysis of different outcomes and is often used to develop cost ranges  and risk reserves. NASA officials stated that the largest cost driver for the  JWST project is the size of the workforce, which could have been  subjected to a sensitivity analysis; yet, the cost model did not include a  sensitivity analysis that would show how staff increasing or decreasing  over time affects cost. In addition, NASA officials believe that all risks  were sufficiently accounted for when conducting the JCL, however, the  software used to conduct the JCL analysis does not recognize certain  risks that officials had placed on activities in the project schedule and,  therefore, some risks were discarded during the simulation.

The schedule used to conduct the JCL was also summarized at such a  high level that the durations were too long to effectively model the risks.  For example, one of the activities that drove the launch date was over 4  years in duration and should have been broken down further prior to  conducting the simulation. Moreover, the critical path in the JCL schedule  consisted of six level of effort activities all with the same duration of 2,238   Level of effort activities should never be on the critical  days in length.path because support activities should never drive any milestone finish  date. As a result of the schedule used in the JCL not fully meeting best  practices, we question the results of the analysis. Furthermore, the risk of  having to carry the JWST workforce to support the project if delayed was  not included since a sensitivity analysis was not performed. Project  officials report that, instead, risk associated with the workforce was  factored in when establishing cost reserves.

In addition, project officials did not commission an independent cost  estimate, which is considered one of the best and most reliable estimate  validation methods because it shows whether other estimating  procedures produce similar results, and it provides an independent view  of expected program costs that tests the program office’s estimate for  reasonableness.independent cost estimate has an increased risk of being underfunded   An estimate that has not been reconciled with an  because the independent cost estimate provides an objective and  unbiased assessment of whether the project estimate can be achieved.  Notably, however, project officials provided evidence that an independent  cost assessment was done for the project at the request of the JWST  Standing Review Board, the independent review team for the project, and  the assessment was within 2 percent of the project’s estimated cost for  the rebaseline. Project officials contend that the approach they used in  developing the life-cycle cost estimate for the project is more accurate  than the types of approaches often used to develop and independent  estimate.
subsections: section_title: Complete Schedule Not Finalized; Full Assessment of Schedule Needed
paragraphs: We did not conduct a full schedule assessment to determine the reliability  of the revised schedule based on best practices due to on-going contract  negotiations. The project has an integrated master schedule developed  as part of the replan; however, it is not finalized because major contract  modifications have yet to be negotiated and definitized. Specifically, the  modification to the Northrop Grumman contract, which accounts for  approximately 40 percent of the total project cost and spans much of the  work on the spacecraft and OTE, remains undefinitized more than a year  after the project was rebaselined. Once the project completes  negotiations for the contract modification and all schedule dates are set,  the project can then have a measurable integrated master schedule.  Project officials stated that the negotiation process and updating of  associated schedules are planned to be complete in January 2013 for the  Northrop Grumman contract modification—a year after submission of the  latest update to its proposal for the replan. The project also reported that  multiple audits of the proposals submitted by Northrop Grumman and its  subcontractor by the Defense Contract Audit Agency have delayed  definitization. Negotiations for the modification to NASA’s contract with  the Space Telescope Science Institute to incorporate the October 2018  launch readiness date are not scheduled to be complete until spring  2013.

Once all the contracts have been definitized and the project’s integrated  master schedule is baselined, we plan to conduct a comprehensive best  practices assessment of the reliability of the project’s schedule estimates.

section_title: Technically Challenging JWST Project Lacks the Schedule Reserve Flexibility and Commensurate Oversight of Integration and Test Efforts
paragraphs: Project officials report that the JWST schedule has 14 months of reserve,  which meets Goddard guidance for schedule reserve; however, only 7 of  the 14 months are likely to be available for the last three of JWST’s five  complex integration and test efforts. GAO’s prior work shows that it is  during integration and test where problems are commonly found and  schedules tend to slip. Given that JWST has a challenging integration and  test schedule, this could particularly be the case. The project has made  some significant progress in the past year, notably successfully  completing development of the 18 primary mirror segments—considered  JWST’s top technical risk. Nevertheless, ongoing challenges are  indicative of the kinds of issues that can require a significant amount of  effort to address. For example, instrument challenges have delayed the  first integration and test effort. In addition, key long-term risks on  subsystems with a significant amount of work remaining will not be retired  until 2016. Currently, NASA’s plan for project oversight calls for one  independent system integration review about 13 months before launch.  While this is consistent with what NASA requires for its projects, this  approach may not be sufficient for a project as complex as JWST. As a  result, the current plan may be inadequate to ensure key technical and  management issues are identified early enough to be addressed within  the current integration and test phase schedule.
subsections: section_title: Test and Integration Schedule Lacks Schedule Reserve Late in the Process
paragraphs: JWST has a complex and lengthy integration and test phase, which  includes five major integration and test efforts—ISIM, OTE, OTIS,  spacecraft, and observatory. See figure 4 for the project reported dates  for the major integration and test efforts and the schedule reserve  allocated for each effort.

Overall, project officials report that the critical path schedule has 14  months of reserve with 7 months after the ISIM and OTE integration and  test efforts. If these efforts are delayed beyond those 7 months, they will  impinge on the schedule for the remaining three integration and test  efforts. Project officials stated that the baseline plan is for the OTIS  integration and test effort to not begin earlier than May 2016. These  officials reported it is likely that all of the 7 months of schedule reserve  held by the OTE subsystem will be utilized during its integration and test  prior to delivery to OTIS and that the OTE effort is on the critical path for  the project. Therefore, the remaining integration and test efforts—OTIS,  Spacecraft, and Observatory—will likely have at most 7 months divided  among them to use if issues are found during integration and test.

In addition to not likely being able to conserve any of the unused first 7  months of schedule reserve, the project has limited time allocated to the  final three integration and test efforts, with between 2 to 4 months for  each. This time could be used easily by the project if an issue were to  arise during integration and test. An example of this is seen in the OTIS  integration and test schedule, which currently has 3 months of schedule  reserve. The final event in the OTIS integration and test effort is a lengthy  cryo-vacuum test—the first time that the optics integrated with the  instruments will be tested at operational temperatures near absolute zero  (less than -400 degrees Fahrenheit)—that takes approximately 3 months,  due to the requirements of the test. If an issue were to arise during this  test that requires shutting the test down and working on the hardware, the  chamber would have to be slowly warmed to a temperature safe for  removal of the hardware from the chamber, work would be performed,  and the 3-month test process would need to begin again. This could  easily exhaust the available schedule reserve. Prior GAO work shows  that it is during integration and test when problems are commonly found,  and schedules tend to slip. A project official confirmed that this is the case  because during integration and test the process is more sequential and  there is less flexibility to move work around if problems are found. A  NASA Inspector General report on the Mars Science Laboratory, another  complex and high-cost mission, found that historically the probability that  schedule-impacting problems will arise is commensurate with the  complexity of the project. JWST is one of NASA’s most technologically  complex projects to date.

section_title: Despite Overcoming Several Technical Challenges, JWST Has a Significant Amount of Work with Complex Technical Challenges Remaining
paragraphs: The project has made significant progress overcoming several technical  challenges over the last year. In December 2011, for example, the project  completed development of the 18 segments of the primary mirror—the  project’s primary technology risk—approximately 6 weeks ahead of  schedule. In addition, project officials stated that during the last year they  were also able to accelerate other optics-related work, which added one  month of funded reserve to the schedule, bringing the total to 14 months.  Finally, the project successfully addressed an increase in the estimated  amount of heat on the instruments, which otherwise could have pushed  observatory temperatures close to where the optics would not function  correctly.

Although technical challenges are being overcome, the project will likely  continue to experience additional challenges over the remainder of the  project, given the significant portion and complexity of the work  remaining. Four of six major subsystems have nearly 50 percent or more  of their development work remaining based on its current budget  information, although the dollar amounts associated with the work vary.  See figure 5.

Currently, the project is experiencing several technical issues that have  required a significant amount of time and effort to address. For example,  the spacecraft subsystem, which experienced delays in development prior  to the replan, is currently estimated to be heavier than its mass limit.Spacecraft development has lagged behind other subsystems because it  was viewed as a lower risk part of the project and was therefore not  allocated funding when budgets were limited prior to the replan. In March  2010, the project passed its mission critical design review, which  evaluated the project design and its ability to meet mission requirements  and indicated that the design was ready for fabrication phase; however,  the spacecraft was not included in this review due to its delayed  development. Under the initial replan, which had constrained funding in  fiscal years 2011 and 2012, the spacecraft critical design review was  scheduled for June 2014; however, due to additional funding in the final  agency-approved replan, the project was able to accelerate work and this  review is now planned for December 2013.

Project officials have been concerned with the mass of JWST since its  inception because of the telescope size and the limits of available launch  vehicles. Accordingly, mass limits have been allocated for each  subsystem, including the spacecraft. Project officials stated that they  expected to encounter mass growth on the spacecraft, but that the  magnitude of the mass growth on the spacecraft was unexpected. As  shown in figure 6, the current spacecraft projected mass exceeds its  mass allocation.

Primary drivers of the mass growth on the spacecraft are increases in the  estimated weight of the wiring harnesses, which distribute power and  electric signals between different parts of the observatory, the solar array,  and other structures that make up the spacecraft. The burden to find  ways to reduce mass has been primarily placed with the spacecraft  because it was assessed by the project to have the least technical risk  and because it is the least mature subsystem and can more easily  accommodate design changes. Over 100 kilograms, or 220 pounds, of  mass savings options are being evaluated by the project and Northrop  Grumman, which is developing the spacecraft. Potential mass solutions  have been identified by Northrop Grumman and the project; however,  cost and risk vary with each solution and the project is still evaluating the  trade-offs of the various solutions. Project officials stated that final  decisions for all tradeoffs will need to occur before spacecraft critical  design review in December 2013.

The ISIM subsystem is experiencing technology and engineering  challenges that resulted in the use of 18 of ISIM’s 26 months of schedule  reserve. The schedule for the instruments needed for ISIM continues to  slip, which could result in use of more schedule reserve. Based on the  replan, all four instruments were to be delivered by September 2012;  however, only two instruments were delivered by that time and those still  have issues that must be addressed. The remaining two instruments are  currently scheduled to be delivered at least 11 months late. See table 1  below for the instrument specific issues.

In addition to the instrument delays, two other technical challenges  associated with ISIM are: (1) the detectors used by three of the four   instruments to capture infrared light in space are degrading and may  need to be replaced, resulting in the addition of another round of cryo- vacuum testing—in which a test chamber is used to simulate the near  absolute zero temperatures in space, and (2) issues with the  development of the cryo-cooler system that removes heat and cools MIRI.  In December 2010 the project became aware that the detectors in three  of the instruments were degrading.million and 15 months of schedule reserve to replace the detectors were  included in the replan. These additions covered the cost of manufacturing  the detectors; fabrication, assembly, and test of new focal plane  assemblies; changing the detectors on three instruments, and the  addition of a third ISIM cryo-vacuum test. The manufacturing process for  new detectors takes approximately 30 months, which means that they  cannot be delivered until after the second round of ISIM cryo-vacuum  testing in 2014. As a result, $2 million of the $42 million in the replan was  used to add a third round of cryo-vacuum testing for ISIM. The third test  will validate the performance requirement of the ISIM and is the only time  the instruments are tested with the flight detectors. Changing the  detectors requires disassembling the instruments from ISIM, a process  that will risk damage to the structure and instruments. Project officials  stated that they will continue to monitor the degradation rate of the current  detectors because if the degradation rate is low, they may not replace the  detectors.

As a result, approximately $42  Development issues with a part of the cryo-cooler needed for MIRI have  delayed its delivery to ISIM. In 2010, project officials realized that an  essential valve in the cryo-cooler was leaking at rates that exceeded  requirements. Following the results of a failure review board, the  contractor manufactured a newly designed valve, but it also did not meet  leak rate requirements. Project officials stated that a new valve design will  not be manufactured in time for use in the first ISIM cryo-vacuum test.  The project is concurrently developing three alternatives and authorized  manufacturing for one of the alternatives in October 2012. Project officials  stated that the MIRI cryo-cooler is particularly complex because it spans  approximately 10 meters—or approximately 33 feet—through the entire  JWST observatory. These issues combined required the use of 18  months of schedule reserve, which reduced ISIM’s schedule reserve from  the 26 months established in the replan to 8 months before it is needed  for integration with the OTIS subsystem.

These types of issues are not uncommon among NASA programs as  technical issues tend to arise when disparate parts are integrated and  tested together for the first time. Given the complexity and cutting edge  technology developed and used on JWST, it is expected that these kinds  of issues will continue to materialize as the program moves through its  complex integration and test program. Figure 7 shows the delay of  instrument deliveries as well as changes to the ISIM integration and test  and final delivery dates over the last year.

Until the project is able to overcome the major issues with the instruments  and other parts of the ISIM, it is likely that the schedule would continue to  slip and may begin to affect the overall project schedule. ISIM still has 8  months of schedule reserve before the slipping of its schedule would  affect the schedule for the remainder of the project. The instrument,  detector, and cryo-cooler issues have all contributed to the delay in the  ISIM integration and test schedule and the reduction of objectives that  can be achieved in the first two rounds of cryo-vacuum testing. The first  round of testing will not include two instruments, a final design of the cryo- cooler hardware, or new detectors. As a result project officials will only be  able to gather risk reduction information on the FGS/NIRISS, MIRI, test  procedures, and test support equipment from the first cryo-vacuum test.

The project also has several known long term risks and challenges  remaining. For example, risks related to OTIS, the sunshield, and the  ground system subsystems are not scheduled to be addressed until late  in project development. As of October 2012, seven of the top 10 project  risks were related to the long-term risks associated with the OTIS and  sunshield, most of which will not be resolved until 2016 or later. For  example, several risks relating to OTE are not scheduled to be closed  until the OTIS testing in the chamber at Johnson Space Center in  February 2017. Project officials are adding risk mitigation through early  and additional testing, where possible, to these subsystems. Prior to the  replan, the ground system software was at high risk for not being  completed before launch and many tasks were planned for completion  after launch. Space Telescope Science Institute officials stated that the  replan allows them to plan for completion of their work before launch on a  more realistic time schedule, which decreases schedule and operational  risk. A continuing challenge on the ground system is that some  development and testing is dependent on the final design of subsystems  such as the instruments, which continue to slip delivery dates.

section_title: Independent and Management Oversight during Test and Integration May Not Be Commensurate with JWST Test and Integration Complexity
paragraphs: The project plans to hold independent and management reviews required  for all projects during the integration and test phase, but this phase for  JWST is particularly complex. JWST has five major integration and test  efforts that span 7 years and only one independent mission-level  technical review—the system integration review. The system integration  review evaluates the readiness of the project and associated supporting  infrastructure to begin system assembly, integration, and test, and  evaluates whether the remaining project development can be completed  within available resources. For JWST, this review is scheduled in  September 2017, only 13 months prior to launch. Projects we reviewed  that had recently launched, however, held their system integration review  on average approximately 22 months prior to launch. The project has an  internal review with participation from standing review board members  planned before the beginning of OTIS integration and test activities begin,  and it will be subject to independent lower level reviews conducted by the  Goddard Systems Review Office of the integration and test process. In  addition, key decision point D (KDP-D)—when the senior agency decision  authority would approve the project to proceed into the system integration  and test phase—is scheduled for December 2017, 3 months after the  commencement of the final major integration and test activity. According  to NASA policy, this review should be held prior to the start of the system  integration and test phase of the project.shows that over 90 percent of expected integration and test funding will  be spent on four major integration and test activities prior to the  scheduled mission-level system integration review and KDP-D approval  by NASA senior management. As a result, the current plan may be  inadequate to ensure that key technical and management issues are  identified early enough to be addressed within the current integration and  test phase schedule.

section_title: JWST Project Has Taken Steps to Enhance Communications with and Oversight of Its Contractors
paragraphs: The JWST project has taken steps to improve communications and  oversight of its contractors as part of the replanning activities. For  example, based on recommendations from the ICRP, the project has  instituted meetings at various levels throughout NASA and its contractors  and subcontractors. In addition, the project has added personnel at  contractor facilities, which has allowed for more direct interaction and  quicker resolution of issues. The project also assumed responsibility of  the mission-level systems engineering function from Northrop Grumman,  a move that shifts the authority to make trades or decisions to NASA. An  independent NASA review of the project conducted in May 2012 found,  however, that agencywide reductions in travel budgets have put the  effectiveness of the JWST project’s oversight plans in jeopardy. While the  project received partial relief from travel budget reductions in fiscal year  2012, project officials are concerned that the current level of oversight will  not be sustained if similar cuts in travel funding occur in future years as  anticipated. The project is also taking steps to enhance its oversight of  project risks by implementing a new risk management system. The new  project manager found that the previous system lacked rigor and was  relatively ineffective for managing project risks, especially for a project as  complex as JWST. The new system should allow for better tracking of  risks than did the previous system. While these enhancements to the  oversight of the project are steps in the right direction, it will take time to  assess their effectiveness.
subsections: section_title: NASA Responded to Recent Independent Review Panel Report with Increased Communication and Oversight
paragraphs: Based on recommendations in the ICRP report, NASA has taken action to  enhance oversight and communications. See table 2 for the ICRP  recommendations and actions taken by NASA in response.

NASA has taken steps to increase communication between the project  and its contractors and subcontractors in an effort to enhance oversight.  According to project officials, the increased communication has allowed  them to better identify and manage project risks by having more visibility  into contractors’ activities. The project reports that a great deal of  communication existed across the project prior to the ICPR and replan;  however, improvements have been made. For example, monthly  meetings between project officials at Goddard and all of the contractors  have continued on a regular basis and include half-day sessions devoted  to business discussions. The project reports that these meetings have  benefits over other forms of communication. For example, it was through  dialogue with several technical leads at Northrop Grumman during  detailed reviews of analytical models that the project identified that the  mass issue on the spacecraft was likely to occur.

In addition, the project has increased its presence at contractor facilities  as necessary to provide assistance with issues. For example, the project  has had two engineers working on a recurring basis at Lockheed Martin  to assist in solving problems with the NIRCam instrument. The ISIM  manager said that these engineers have insight into Lockheed Martin’s  work and are having a positive effect as they offer technical help and are  involved in devising the solutions to issues. He added that that these  engineers have authority to make decisions on routine issues to allow the  work flow to continue, but decisions that are more complex or require a  commitment of funds are communicated to project management for  disposition. The project reports that the Jet Propulsion Laboratory,  responsible for NASA contribution to the MIRI instrument and its  associated cryo-cooler, has an in-house representative in the responsible  Northrop Grumman division to monitor the work being performed on the  cryo-cooler.

The JWST project also assumed full responsibility for the mission system  engineering functions from Northrop Grumman in March 2011. NASA and  Northrop Grumman officials both said that NASA is better suited to  perform these tasks. Project officials stated the systems engineering  requires the ability to make trades and decisions across the entire  observatory, and because Northrop Grumman is only responsible for  portions of the observatory, it did not have the authority to make trades or  decisions for areas outside of its control. Although responsibility for the  overall mission systems engineering function was removed from Northrop  Grumman, it retains system engineering responsibility for work still under  its contract, such as development of the spacecraft and sunshield. The  ICRP noted that a highly capable, experienced systems engineering  group is fundamental to project success and appropriate to ensure  accountability especially for a project of JWST’s complexity and visibility.

While these enhancements to the oversight of the project are steps in the  right direction, it will take time to assess their effectiveness. In addition,  sustainment of these efforts on the part of the project will be important.  Project and contractor officials we spoke with believe that the increased  communication has had a positive effect on the relationships between  them. We will continue to monitor the interaction between the project and  its contractors and its frequency in future reviews to identify whether the  changes have had the desired results.

section_title: Travel Budget Reductions May Hamper Planned Oversight Activities
paragraphs: The JWST project reported that its travel budget was reduced by  approximately $200,000 from the $1.2 million planned in fiscal year 2012  as a result of NASA’s implementation of an Executive Order to promote   According to project officials, the changes in  more efficient spending.oversight necessitated by a reduction in travel funds represent a major  shift away from the management paradigm adopted during the replan.  Proposed reductions in future fiscal years could significantly reduce the  project’s travel budget. The project reports that the travel requirements for  fiscal years 2013 through 2015 are $1.6 million, $1.7 million, and $1.8  million, respectively. Officials reported that while travel is a small  percentage of the project’s annual budget, the majority of expected  travel—about 87 percent—is for oversight functions put in place as a  result of the ICRP recommendations, such as having a permanent on-site  presence at Northrop Grumman. These oversight functions include  attending and participating in contractor monthly programmatic and  technical reviews, technical interface meetings, recurring on-site  presence at contractor facilities for quality assurance reviews and  inspection of hardware. JWST project officials are concerned that  decreased oversight could translate into the project increasing its use of  cost and schedule reserves as they will not be conducting planned  oversight to better ensure success. A recent NASA Office of Evaluation  review concluded that by not having an adequate travel budget, the  project is at risk of cost/schedule growth and/or technical risk due to the  late identification of issues or timely resolution strategies. The project has  made adjustments to absorb the reduction in fiscal year 2012 and plans  to identify instances of increased cost or schedule risk due to late  identification of issues. However, the project does not have a strategy to  address anticipated future reductions. Ensuring adequate oversight is  particularly important as the project begins its complex and lengthy test  and integration phase, where issues will likely surface.

section_title: NASA Headquarters Independent Review of the JWST Program Identified Concerns and Recommends Indicators to Measure Progress
paragraphs: As part of NASA’s approach to increase oversight of the project at  headquarters, NASA’s Office of Evaluation recently conducted an  independent review of the JWST project to assess the progress since the  September 2011 rebaseline was approved. According to the Director of  the Office of Evaluation, the goal of the review was not to reproduce the  replan assessment, but rather to assess progress based on cost,  schedule, and technical performance of the project and the status of  oversight functions within NASA headquarters, the JWST Program Office,  and Goddard Space Flight Center. The intended outcome of the review  was 1) to obtain a snapshot of performance to determine if the program  was progressing in accordance with its plan, and 2) to identify leading  indicators for upper management to use when tracking future  performance. The review team identified several areas of concern within  the program, many of which we have highlighted, and recommended a list  of leading indicators that project management should consider tracking.

The Director of the Office of Evaluation said that the project is generally  performing the activities and maintaining the schedule set forth in the  replan; however, the team identified key areas that should be monitored  as the project moves forward. The review team also recommended a set  of leading indicators for project management to consider tracking to  measure and monitor progress. The Director added that these indicators  are for the project to use and would not be specific criteria for use by  independent review boards such as the Standing Review Board. These  indicators are a positive step to ensure that NASA management has the  information necessary to monitor the progress of the JWST project. See  table 3 below for the concerns raised by the review team.

The new JWST project manager re-emphasized the importance of the  project’s risk management system and, in August 2012, a new risk  management database was implemented to support the system. The  project manager told us that he evaluated the risk management system  being utilized by the project when he assumed his position and found it to  be ineffective and not robust, especially for a project as complex as  JWST. While the basic risk management methodology remains  unchanged, the project manager wanted a more regimented system. For  example, the project utilizes a hierarchy of risk boards that periodically  reviews and provides disposition of all new and existing risks. These risk  boards reviewed and assessed new risks and lower level risk board  actions and met on an ad hoc basis. The project manager instituted a  more regimented system that re-emphasized and revised the weekly  project risk board meetings. Lower level risk boards meet a minimum of  once a month depending on activity.

The project manager also determined that a new risk management  database needed to be put in place that would bring more rigor to the risk  management process. The project manager told us that he directed an  overhaul of the risk management database to provide more complete  information to management on the purpose and history for each risk. The  goal was to improve consistency in how the project determined the  potential for a risk to occur and its impact, and provide greater detail on  mitigation and better tracking of the status for each risk. For example, the  new system puts more emphasis on understanding and capturing the key  events in the mitigation plan that are intended to result in a change in  likelihood or consequence of a risk. The new system has a provision  where the mitigation plan will be entered and updated over time, and the  capability to store data such as mitigation steps throughout the life of the  risk. In addition, the new system now archives data automatically to  provide a traceable history of the risk. The prior data system did not have  as robust of an archiving function. Furthermore, the project manager  wanted to improve the linkage between the risk database entries and  financial records to ensure consistency of the data in the risk database  with regard to cost and schedule for risk mitigations with project office  financial records. As the changes to the risk management system and  database, as well as other changes we identified that were put in place to  enhance oversight were just recently implemented, we will continue to  monitor their continued use and assess the impact they may be having on  the project.

section_title: Conclusions
paragraphs: The JWST project is among the most challenging and high-risk projects  NASA has pursued in recent years. It is also one of the most expensive,  with a recent major replan resulting in a total cost of $8.8 billion. The  reasons for cost and schedule growth were largely recognized by an  independent review team to be rooted in ineffective funding,  management, communication, and oversight. NASA has invested  considerable time and resources replanning the project and instituting  management and oversight improvements in order to ensure that it (1)  can be executed within its new estimates and (2) has addressed the  majority of issues raised in the recent independent review. It appears that  communications with contractors and within NASA have improved, that a  more robust risk mitigation system is in place, that more is known about  what it will take to complete the project and how much it will cost, and that  the project is currently meeting the majority of its milestones.  Nevertheless, over the course of the next several years, the project will  be executing a large amount of work with several extremely complex and  challenging integration and test efforts. Because three major test and  integration efforts must be completed in the last 2 years of the JWST  schedule, it is essential that issues are identified and addressed early  enough to be handled within the project’s current schedule. While the  JWST oversight plan is consistent with NASA’s requirements for all  project’s required reviews, a single independent review scheduled just  over a year before launch may not be sufficient to identify and resolve  problems early for a project of this magnitude. A key element of  overseeing project progress is monitoring how the project is executing to  its cost baseline. To that end, while NASA took some steps that were in  line with best practices to develop its revised baseline, some of the  deficiencies we found in its process could impact the reliability of the cost  estimate and the joint cost and schedule confidence level that was  provided to headquarters decision-makers. Without higher-fidelity,  regularly updated information related to costs, as well as an oversight  regime during later phases of test and integration that is commensurate  with the complexity of that effort, NASA risks late identification of  technical and cost issues that could delay the launch of JWST and  increase project costs beyond established baselines. Also important to  oversight for the remainder of the project is the ability of officials to  sustain improvements to communication with and oversight of  contractors. Anticipated travel restrictions, however, could decrease the  project team’s ability to sustain these actions. Without a plan to address  such reductions in future years, the project could once again become  susceptible to communication and oversight problems identified in earlier  reviews, which could also have a detrimental impact on continued project  performance.

section_title: Recommendations for Executive Action
paragraphs: To ensure that the JWST life-cycle cost estimate conforms to best  practices, GAO recommends that the NASA Administrator direct JWST  officials to take the following three actions to provide high-fidelity cost  information for monitoring project progress:  improve cost estimate documentation and continually update it to  reflect earned value management actual costs and record any  reasons for variances,  conduct a sensitivity analysis on the number of staff working on the  program to determine how staff variations affect the cost estimate,  and perform an updated integrated cost/schedule risk analysis, or joint  cost and schedule confidence level analysis, using a schedule that  meets best practices and includes enough detail so that risks can be  appropriately mapped to activities and costs; historical, analogous  data should be used to support the risk analysis.

To ensure that technical risks and challenges are being effectively  managed and that sufficient oversight is in place and can be sustained,  GAO recommends that the NASA Administrator direct JWST officials to  take the following three actions:  conduct a separate independent review prior to the beginning of the  OTIS and spacecraft integration and test efforts to allow the project’s  independent standing review board the opportunity to evaluate the  readiness of the project to move forward, given the lack of schedule  flexibility once these efforts are under way,  schedule the management review and approval to proceed to  integration and test (key decision point D or KDP-D) prior to the start  of observatory integration and test effort, and devise an effective, long-term plan for project office oversight of its  contractors that takes into consideration the anticipated travel budget  reductions.

section_title: Agency Comments and Our Evaluation
paragraphs: NASA provided written comments on a draft of this report. These  comments are reprinted in appendix IV. NASA also provided technical  comments, which were incorporated as appropriate.

In responding to a draft of this report, NASA concurred with three  recommendations and partially concurred with three other  recommendations and commented on actions in process or planned in  response. In some cases, these actions meet the intent and are  responsive to issues we raise; however, some of the responses do not  fully address the issues we raised in the report.

NASA partially concurred with our recommendation to improve the cost  estimate documentation of the JWST project, and to continually update it  to reflect earned value management actual costs and record any reasons  for variances between planned and actual costs. In response to this  recommendation, NASA officials stated that the project currently receives  earned value data from some of its contractors and performs monthly  analysis of that data to understand the contractors’ estimates at  completion, and then compares these numbers to similar figures  independently assessed by the JWST project. NASA also highlighted its  efforts to improve the agency’s documentation of the earned value  variances and to extend the earned value management analysis to areas  where it is not yet implemented, such as ground systems development at  the Space Telescope Science Institute. In addition, NASA responded that  its annual budget process generates a requirements-driven budget plan  consistent with the rebaseline. NASA stated that this information is  updated in several different documents that are provided to management  and it does not plan to revise its JCL documentation developed during the  replan. Despite these steps, we could not independently confirm that they  were leading to an updated cost estimate, which is the basis of our  recommendation. If the estimate is not updated, it will be difficult to  analyze changes in project costs and collecting cost and technical data to  support future estimates will be hindered. Furthermore, if not properly  updated on a regular basis, the cost estimate cannot provide decision  makers with accurate information for assessing alternative decisions.  Without a documented comparison between the current estimate  (updated with actual costs) and the old estimate, the cost estimator  cannot determine the level of variance between the two estimates and  cannot see how the project is changing over time. Therefore, we continue  to believe NASA will be well served by following best practices and  updating its cost estimate with current information and documenting  reasons for any variances. We encourage the project to improve the cost  estimate documentation and record any reasons for variances between  planned and actual costs and we intend to review the documentation as a  part of our ongoing review of the project.

NASA officials partially concurred with our recommendation that the  project conduct a sensitivity analysis on the number of staff working on  the project to determine how staff variations affect the cost estimate. In its  response, the agency stated that it believes it met the intent of this  recommendation when staffing levels were determined in the 2011 JWST  rebaseline based on programmatic experience from the accomplishment  of similar activities. To accommodate the possibility of increased costs  based on increased staffing hours, NASA reports that funded schedule  reserve was built into the JWST rebaseline, in addition to unallocated  future expenses being held at various levels of the organization. NASA  believes that these reserves will be sufficient to cover increases for the  duration of specific activities that result in increased staffing cost, and that  an additional workforce sensitivity analysis is not warranted. NASA added  that the joint cost and schedule confidence level analysis performed  provided a de facto workforce sensitivity analysis and does not plan any  further action. A joint cost and schedule confidence level analysis,  however, is not the same as a sensitivity analysis wherein the sources of  the workforce variation should be well documented and traceable. While  we appreciate the steps NASA took to account for workforce variation, the  JWST cost model does not show how staff levels increasing or  decreasing over time affects cost. Furthermore, best practices call for a  risk analysis to be conducted in conjunction with a sensitivity analysis, not  to be a substitute for it. As a best practice, a sensitivity analysis should be  included in all cost estimates because it examines the effects of changing  assumptions and ground rules. Since uncertainty cannot be avoided, it is  necessary to identify the cost elements that represent the most risk and, if  possible, cost estimators should quantify the risk. Without performing a  sensitivity analysis that reveals how the cost estimate is affected by a  change in a single assumption, such as workforce size, the cost estimator  will not fully understand which variable most affects the cost estimate.  Therefore, we continue to believe that NASA should conduct a sensitivity  analysis for the JWST project, given the large number of staff working on  the program, to determine how staff variations positively or negatively  affect the cost estimate rather than relying on schedule reserve and  unallocated future expenses to offset any shortfall.

NASA concurred with our recommendation to perform an updated  integrated cost and schedule risk analysis using a schedule that meets  best practices and includes enough detail so that risks can be  appropriately mapped to activities and costs. In response to this  recommendation, NASA stated that the agency is already using tools and  a method to conduct programmatic assessments of projects after the  baseline was established using the JCL methodology. While these may  be good tools, the key point is the need to address shortcomings of the  schedule that supports the baseline itself. For example, the lack of detail  in the summary schedule used for the joint cost and schedule risk  analysis prevented us from sufficiently understanding how risks were  incorporated; therefore, we question the results of that analysis. Since the  JCL was a key input to the decision process of approving the project’s  new cost and schedule baseline estimates, we maintain that the JWST  project should perform an updated JCL analysis using a schedule with  sufficient detail to map risks to activities and costs. Doing so could help  increase the reliability of the cost estimate and the confidence level of the  JCL. Furthermore, risk management is a continuous process that  constantly monitors a project’s health. Given that JWST is many years  from launch and the risks that the project faces are likely to change, a risk  analysis should be conducted periodically throughout the life of the  project.

NASA concurred with our recommendation to conduct a separate  independent review prior to the beginning of the OTIS and spacecraft  integration and test efforts. In response to this recommendation, NASA  stated that it will request members of the independent JWST Standing  Review Board participate in OTIS Pre-Environmental Review scheduled  prior to the beginning of OTIS environmental testing. A member of the  Standing Review Board will co-chair this review and report its findings to  the NASA Associate Administrator, which is the practice of all Standing  Review Board reviews. In addition, NASA plans to direct Northrop  Grumman, the spacecraft developer, to add members of the Standing  Review Board, as well as members of the Goddard Independent Review  Team, to the spacecraft element integration readiness review and report  their findings to the NASA Associate Administrator. We believe these  actions meet the intent of our recommendation and will afford an  independent evaluation of the readiness of the project to move forward  with its major integration and test efforts.

NASA partially concurred with our recommendation to schedule the  management review and approval to proceed to integration and test  (KDP-D) prior to the start of the observatory integration and test effort. In  response to this recommendation, NASA stated that it will reduce the 3- month gap between the scheduled system integration review and the  KDP-D review, which it believes will provide NASA management and the  NASA Associate Administrator with the full independent assessment  earlier than currently planned. While we agree that this change will move  the review earlier than previously planned, based on its response, NASA  still plans to hold the review after the observatory integration and test is  already underway. Holding this review after the observatory integration  and test effort is already underway does not meet agency policy and will  lessen the impact of the review as it may be inadequate to ensure key  technical and management issues are identified early enough to be  addressed. KDP-D is the point in which management approval is given to  transition to the test and integration phase. We reiterate our  recommendation that NASA should hold this important key decision point  prior to the beginning of this last major integration and test effort, as  required by agency policy.

NASA concurred with our recommendation to devise an effective, long- term plan for project office oversight of its contractors that takes into  consideration the anticipated travel budget reductions. In response to this  recommendation, NASA stated that it will develop a plan based on fiscal  year 2013 travel allocations and will take into consideration anticipated  travel budget reductions. In addition, NASA stated that the plan will  enable the project to maintain oversight of JWST contractors and their  ability to meet performance and delivery deadlines and work closely with  the international partners. We believe such a plan will be critical to  ensuring adequate oversight, which is particularly important as the project  enters into the complex integration and test efforts where issues will likely  surface. In addition, we agree with the concerns of project officials that  the current efforts to increase communication and oversight may not be  sustained if reductions to future travel budgets occur as anticipated. We  encourage the project to complete this plan in a timely manner and intend  to review it as a part of our ongoing assessment of the project’s oversight  efforts.

We will send copies of the report to NASA’s Administrator and interested  congressional committees. We will also make copies available to others  upon request. In addition, the report will be available at no charge on  GAO’s web-site at http://www.gao.gov.

Should you or your staff have any questions on matters discussed in this  report, please contact me at (202) 512-4841 or chaplainc@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix V.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to assess (1) the extent to which NASA’s revised  cost and schedule estimates are reliable based on GAO best practices,  (2) the major risks and technological challenges the James Webb Space  Telescope (JWST) project faces, and (3) the extent to which the National  Aeronautics and Space Administration (NASA) has improved the  oversight of the JWST project. In assessing the project’s cost and  schedule estimates, we performed various checks to determine that the  provided data were reliable enough for our purposes. Where we  discovered discrepancies, we clarified the data accordingly. Where  applicable, we confirmed the accuracy of NASA-generated data with  multiple sources within NASA.

Northrop Grumman, the Space Telescope Science Institute, and the  JWST program and project offices. After reviewing cost estimate  documentation submitted by NASA and conducting numerous interviews  with relevant sources within the project office, we calculated the  assessment rating of each criteria within the four characteristics by  assigning each individual assessment rating: Not Met = 1, Minimally Met  = 2, Partially Met = 3, Substantially Met = 4, and Met = 5. We then took  the average of the individual assessment ratings for the criteria to  determine the overall rating for each of the four characteristics. The  resulting average becomes the “Overall Assessment” as follows: Not Met  = 1.0 to 1.4, Minimally Met = 1.5 to 2.4, Partially Met = 2.5 to 3.4,  Substantially Met = 3.5 to 4.4, and Met = 4.5 to 5.0. We discussed the  results of our assessments with officials within the program office at  NASA headquarters and the project office at Goddard Space Flight  Center.

We supplemented the assessment of the revised 2011 cost estimate with  an assessment of the summary schedule used for the JCL, which was a  part of the project’s cost estimation process, and followed criteria laid out  in the GAO schedule guide. These practices address whether the  schedule (1) captured all activities; (2) sequenced all activities—that is,  listed in the order in which they are to be carried out; (3) assigned  resources to all activities; (4) established the duration of all activities; (5)  integrated schedule activities horizontally and vertically, which identifies  whether products and outcomes associated with other sequenced  activities are arranged in the right order, and that varying levels of  activities and supporting subactivities are also aligned properly; (6)  established for all activities, the critical path, which is the longest  continuous sequence of activities that is necessary to examine the effects  of activities slipping in the schedule; (7) identified between activities float,  which is the amount of time by which a predecessor activity can slip  before the delay affects the program’s estimated finish date; (8) identified  a level of confidence using a schedule risk analysis; and (9) was updated  using logic and durations to determine dates. We also reviewed the inputs  to the JCL model, the document outlining the methodology of the analysis  that accompanied the electronic files, and interviewed cognizant project  officials to discuss their use of the summary schedule.

Because the project’s detailed integrated master schedule has not been  finalized because of ongoing negotiations and contract modifications, we  did not conduct a complete schedule analysis using the GAO schedule  assessment guide. We plan to perform this assessment in a subsequent  review of the JWST project.

To assess the major short- and long-term risks and technological  challenges facing the project, we reviewed the project’s risk list, monthly  status reviews, and other documentation provided by projects and  contractor officials. This information covered the risks, mitigation plans,  and timelines for addressing risk and technological challenges. We also  interviewed project officials for each major observatory subsystems to  clarify information and to obtain additional information on risks and  technological challenges. Further, we interviewed officials from the Jet  Propulsion Laboratory, Northrop Grumman Aerospace Systems,  Lockheed Martin Advanced Technology Company, Teledyne Imaging  Sensors, the University of Arizona, and the Space Telescope Science  Institute concerning risks and challenges on the subsystems, instruments,  or components they were developing. We reviewed GAO’s prior work on  NASA Large Scale Acquisitions, NASA Office of Inspector General  reports, and NASA’s Space Flight Program and Project Management  Requirements and Systems Engineering Processes and Requirements   We compared NASA’s controls as outlined in these  policy documents.agency policies with the project plan to assess the extent to which the  JWST’s plan followed the intent of the policies with regard to independent  oversight and management approval processes.

To assess the extent to which NASA is performing enhanced oversight of  the JWST project, we reviewed documentation from the Independent  Comprehensive Review Panel and the project to determine actions taken  by NASA in response to the panel’s recommendations. We interviewed  project officials to understand the impact of these changes on the  oversight processes for the project and communication between the  project and its contractors. We also interviewed officials from the Jet  Propulsion Laboratory, Northrop Grumman Aerospace Systems,  Lockheed Martin Advanced Technology Company, Teledyne Imaging  Sensors, the University of Arizona, and the Space Telescope Science  Institute concerning project oversight of work they were performing and  the effectiveness of oversight changes. In addition, we reviewed a  presidential directive and Office of Management and Budget and project  documentation and interviewed project officials concerning the reductions  to travel budgets and their impact on project oversight activities. We  interviewed the Director of NASA’s Office of Evaluation about a recent  internal review of the JWST project and reviewed documentation from  that review. We also reviewed documentation and interviewed project  officials concerning the changes made to the project’s risk management  system.

Our work was performed primarily at NASA Headquarters in Washington,  D.C., and Goddard Space Flight Center in Greenbelt, Maryland. We also  visited Johnson Space Center in Houston, Texas, and the Jet Propulsion  Laboratory in Pasadena, California. In addition, we met with  representatives from Northrop Grumman Aerospace Systems, Lockheed  Martin Advanced Technology Company, Teledyne Imaging Sensors, the  University of Arizona, and the Space Telescope Science Institute.

We conducted this performance audit from February 2012 to December  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

section_title: Appendix II: Subsystems of the JWST Observatory

section_title: Appendix III: Our Evaluation of JWST’s Cost Estimate Process
paragraphs: In determining that the National Aeronautics and Space Administration’s  (NASA) processes for developing the James Webb Space Telescope  (JWST) cost estimate do not fully comply with best practices, we  evaluated the project’s cost estimation methods against our 2009 Cost  Estimating and Assessment Guide. (See table 4.) We applied the  following scale across the four categories of best practices:

Not met: NASA provided no evidence that satisfies any portion of the  criterion.

Minimally met: NASA provided evidence that satisfies less than one- half of the criterion.

Partially met: NASA provided evidence that satisfies about one-half of  the criterion.

Substantially met: NASA provided evidence that satisfies more than  one-half of the criterion.

Met: NASA provided complete evidence that satisfies the entire  criterion.

section_title: Appendix IV: Comments from the National Aeronautics and Space Administration

section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Shelby S. Oakley, Assistant  Director; Karen Richey, Assistant Director; Richard A. Cederholm; Laura  Greifner; Cheryl M. Harris; David Hulett; Jason Lee; Kenneth E. Patton;  Sylvia Schatz; Stacey Steele; Roxanna T. Sun; Jay Tallon; and Jade A.  Winfree made key contributions to this report.


Question-summary annotations: questions: question: What does the current cost estimate of JWST reflect?
answer: Its current cost estimate reflects some features of best practices for developing reliable and credible estimates.
child_questions: question: How specifically did the current cost estimate use best practices?
answer: For example, the estimate substantially meets one of four cost characteristics--comprehensive--that GAO looks for in a reliable cost estimate, in part because all life cycle costs were included.

question: What was the cost estimate lacking?
answer: The estimate, however, only partially met the other three characteristics--well documented, accurate, and credible--which detracts from its reliability.

question: How was the estimate's accuracy and confidence level lessened?
answer: For example, the estimate's accuracy, and therefore the confidence level assigned to the estimate, was lessened by the summary schedule used for the joint cost and schedule risk analysis because it did not provide enough detail to determine how risks were applied to critical project activities.

question: Why was the credibility of the estimate lessened?
answer: The estimate's credibility was also lessened because officials did not perform a sensitivity analysis that would have identified key drivers of costs, such as workforce size.

question: What does GAO believe?
answer: GAO believes there is time to improve the estimate and enhance the prospects for delivering the project according to plan.

questions: question: What is the problem with the amount of reserve JWST has?
answer: Project officials report that the JWST schedule has 14 months of reserve, which meets Goddard guidance for schedule reserve; however, only 7 of the 14 months are likely to be available for the last three of JWST's five complex integration and test efforts. GAO's prior work shows that the integration and test phases are where problems are commonly found and schedules tend to slip.
child_questions: question: How might JWST encounter problems?
answer: GAO's prior work shows that the integration and test phases are where problems are commonly found and schedules tend to slip. Given that JWST has a challenging integration and test schedule, this could particularly be likely.

question: What ongoing challenges does JWST have?
answer: For example, instrument challenges have delayed the first integration and test effort. In addition, key long-term risks on subsystems with a significant amount of work remaining will not be retired until 2016.

question: What does NASA's plan for project oversight call for?
answer: Currently, NASA's plan for project oversight calls for one independent mission-level system integration review about 13 months before launch.

questions: question: How has JWST improved communications of the oversight of the project and its contractors?
answer: JWST has taken several steps to improve communications and oversight of the project and its contractors--such as taking over responsibility for mission systems engineering from the prime contractor; instituting meetings that include various levels of NASA, contractor, and subcontractor management; and implementing a new risk management system to allow for better tracking of risks.
child_questions: question: What is the projected view of the enhancements to the oversight project?
answer: The enhancements to the oversight of the project are steps in the right direction, but it will take time to assess their effectiveness and ensure that the efforts are sustained by the project in the future.

question: What could require the oversight approach to be changed?
answer: Reductions in travel budgets, however, could require the project to adjust the oversight approach that was adopted as a result of the replan.
child_questions: question: How will this problem be addressed?
answer: Additional reductions in travel budgets are anticipated in future years, but officials do not have a plan to address such reductions and their potential impact on continuing the current oversight approach.

questions: question: What is the purpose of JWST?
answer: JWST is one of NASA's most expensive and technologically advanced science projects, intended to advance understanding of the origin of the universe.
child_questions: question: How has the cost of JWST changed?
answer: In 2011, JWST was rebaselined with a life cycle cost estimate of $8.8 billion and a launch readiness date in October 2018--almost nine times the cost and more than a decade later than originally projected in 1999.

question: Why did GAO report on JWST?
answer: Concern about the magnitude of JWST's cost increase and schedule delay and their effects on NASA's progress on other high-priority missions led conferees for the Consolidated and Further Continuing Appropriations Act, 2012, to direct GAO to report on the project.
child_questions: question: What did GAO assess?
answer: Specifically, GAO assessed (1) the extent to which NASA's revised cost and schedule estimates are reliable based on best practices, (2) the major risks and technological challenges JWST faces, and (3) the extent to which NASA has improved oversight of JWST.

question: How did GAO get information for the report?
answer: To do this, GAO compared NASA's revised cost and schedule estimates with best practice criteria, reviewed relevant contractor and NASA documents, and interviewed project and contractor officials.