James Webb Space Telescope: Project Meeting Commitments but Current Technical, Cost, and Schedule Challenges Could Affect Continued Progress
Published: 2014-01-08T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: JWST is one of NASA's most complex and costly science projects. Effective execution of the project is critical given the potential effect further cost increases could have on NASA's science portfolio. The project was rebaselined in 2011 with a 78 percent life-cycle cost estimate increase --now $8.8 billion--and a launch delay of 52 months--now October 2018. GAO has made a number of prior recommendations, including that the project perform an updated cost and schedule risk analysis to improve cost estimates. GAO was mandated to assess the program annually and report on its progress. This is the second such report.
This report assesses the (1) extent to which the JWST project is meeting its cost and schedule commitments and maintaining oversight, (2) current major technological challenges facing the project, (3) extent to which cost risks exist that may threaten the project's ability to execute as planned, and (4) extent to which the JWST project schedule is reliable based on scheduling best practices. GAO reviewed relevant NASA and contractor documents, interviewed NASA and contractor officials, and compared the project schedule with best practices criteria.
section_title: What GAO Found
paragraphs: The James Webb Space Telescope (JWST) project is generally executing to its September 2011 revised cost and schedule baseline; however, several challenges remain that could affect continued progress. The National Aeronautics and Space Administration (NASA) has requested funding that is in line with the rebaseline and the project is maintaining 14 months of schedule reserve prior to its launch date. Performance data from the prime contractor indicate that generally work is being accomplished on schedule and at the cost expected; however, monthly performance declined in fiscal year 2013. Project officials have maintained and enhanced project oversight by, for example, continuing quarterly NASA and contractor management meetings and instituting a tool to update cost estimates for internal efforts. Program officials, however, are not planning to perform an updated integrated cost/schedule risk analysis, as GAO recommended in 2012, stating that the project performs monthly integrated risk analyses they believe are adequate. Updating the more comprehensive analysis with a more refined schedule and current risks, however, would provide management and stakeholders with better information to gauge progress.
The JWST project has made progress addressing some technical challenges that GAO reported in 2012, such as inadequate spacecraft mass margin, but others have persisted, causing subsystem development delays and cost increases. For example, the development and delivery schedule of the cryocooler--which cools one instrument--was deemed unattainable by the subcontractor due to technical issues and its contract was modified in August 2013 for the second time in less than 2 years, leading to a cumulative 120 percent increase in contract costs. While recent modifications have been made, execution of the cryocooler remains a concern given that technical performance and schedule issues persist.
Overall the project is maintaining a significant amount of cost reserves; however, low levels of near-term cost reserves could limit its ability to continue to meet future cost and schedule commitments. Development challenges have required the project to allocate a significant portion of cost reserves in fiscal year 2014. Adequate cost reserves for the prime contractor are also a concern in fiscal years 2014 and 2015 given the rate at which these cost reserves are being used. Limited reserves could require work to be extended or work to address project risks to be deferred--a contributing factor to the project's prior performance issues. Potential sequestration and funding challenges on other major NASA projects could limit the project's ability to address near-term challenges.
GAO's analysis of three subsystem schedules determined that the reliability of the project's integrated master schedule--which is dependent on the reliability of JWST's subsystem schedules--is questionable. GAO's analysis found that the Optical Telescope Element (OTE) schedule was unreliable because it could not adequately identify a critical path--the earliest completion date or minimum duration it will take to complete all project activities, which informs officials of the effects that a slip of one activity may have on other activities. In addition, reliable schedule risk analyses of the OTE, the cryocooler, or the Integrated Science Instrument Module schedules were not performed. A schedule risk analysis is a best practice that gives confidence that estimates are credible based on known risks so the schedule can be relied upon to track progress.
section_title: What GAO Recommends
paragraphs: Congress should consider directing NASA to perform an updated integrated cost/schedule risk analysis. GAO recommends that NASA address issues related to low cost reserves and perform schedule risk analyses on the three subsystem schedules GAO reviewed. NASA concurred with GAO's recommendations.

section_title: Letter
paragraphs: At a current expected cost of approximately $8.8 billion, the James Webb  Space Telescope (JWST) is one of the National Aeronautics and Space  Administration’s (NASA) most complex and costly science projects.  Effective execution of the JWST project is critical given the tradeoffs that  the agency made to ensure that it was well-funded and the potential  impact that any future cost increases or delays could have on the health  of NASA’s overall science portfolio. Specifically, based on the cost and  schedule estimates established in 2011, the cost of the project is almost  nine times the cost in its original plan established in 1999, and the impact  of funding JWST for more years than originally anticipated has  significantly decreased the amount of funds NASA has available to begin  work on other major projects. NASA will have the opportunity over the  next few years, during the course of several very complex test events, to  demonstrate the credibility of the 2011 plan. While additional setbacks are  to be expected given the technically challenging and complex nature of  the project, much is at stake, including the progress NASA has made in  restoring the confidence of Congress and other decision makers in its  ability to effectively plan for and execute major acquisitions.
The on-time and on-budget delivery of JWST is a high congressional  priority. In November 2011, the conference report for the Consolidated  and Further Continuing Appropriations Act, 2012, mandated GAO to  assess the program annually and to report to the Committees on  Appropriations on key issues relating to program and risk management,  achievement of cost and schedule goals, and program technical status.  In our first report, issued in 2012, we assessed the reliability of NASA’s  revised cost estimate based on best practices, the major risks and  technological challenges facing the project, and NASA’s oversight of the  JWST project. This report is our second in response to that mandate. For  this report, we assessed (1) the extent to which the JWST project is  meeting its cost and schedule commitments and maintaining oversight,  (2) the current major technological challenges facing the JWST project,  (3) the extent to which cost risks exist that may threaten the project’s  ability to execute as planned, and (4) the extent to which the JWST  project schedule is reliable based on scheduling best practices.
Our approach included an examination of the overall cost and schedule  progress that NASA has made since our last report in December 2012  and the JWST program changes made in 2011. We examined and  analyzed earned value management data; analyzed the progress made  and any variances to milestones established during the replan in 2011;  and interviewed officials from the JWST program, various contractors,  and the Defense Contract Management Agency to determine the extent  to which oversight was being conducted. To identify the major  technological risks and challenges facing the project, we reviewed the  project’s risk database, monthly status reviews, and other documentation.  We also interviewed project and contractor officials for the major  subsystems. To identify the cost risks that may threaten the project’s  ability to execute as planned, we examined data on JWST’s cost reserve  posture and reviewed project and contractor documentation on risks to  maintaining cost targets and plans to mitigate those risks. We took steps  to confirm the accuracy of the data and performed various checks to  determine that the data provided was reliable enough for our purposes.  To determine the extent to which the JWST project schedule can be  relied on to accurately represent the potential impact of schedule risks  and technical challenges, we analyzed three subsystem schedules that  are used as inputs to the integrated master schedule using scheduling  best practices. We also conducted interviews with project and contractor  management, schedulers, and analyzed project and contractor  documentation concerning scheduling policies and practices. See  appendix I for a detailed description of our scope and methodology.
We conducted this performance audit from February 2013 to January  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: JWST is a large deployable, infrared-optimized space telescope intended  to be the scientific successor to the aging Hubble Space Telescope.  JWST is designed for a 5-year mission to find the first stars and trace the  evolution of galaxies from their beginning to their current formation, and is  intended to operate in an orbit approximately 1.5 million kilometers—or 1  million miles—from the Earth. With its 6.5-meter primary mirror, JWST will  be able to operate at 100 times the sensitivity of the Hubble Space  Telescope. A tennis-court-sized sunshield will protect the mirrors and  instruments from the sun’s heat to allow the JWST to look at very faint  infrared sources. The Hubble Telescope operates primarily in the visible  and ultraviolet regions of the electromagnetic spectrum. The observatory  segment of JWST includes several major subsystems.subsystems are being developed through a mixture of NASA, contractor,  and international partner efforts. See figure 1.
subsections: section_title: Optimal Environment for Instrument to Function
paragraphs: The Mid-Infrared Instrument (MIRI)—one of JWST’s four instruments in  the Integrated Science Instrument Module (ISIM)—requires a dedicated,  interdependent two-stage cooler system designed to bring the optics to  the required temperature of 6.7 Kelvin (K), just above absolute zero. This  system is referred to as a cryocooler. See figure 2 for a depiction of the  cooling system on JWST.
The cryocooler moves helium gas through 10 meters (approximately 33  feet) of refrigerant lines from the sun-facing surface of the JWST  observatory to the colder shaded side where the ISIM is located.  According to NASA officials, a cooler system of this configuration, with so  much separation between the beginning and final cooling components,  has never been developed or flown in space before. Project officials  stated that the MIRI cryocooler is particularly complex and challenging  because of this relatively great distance between cooling components  located in different temperature regions of the observatory and the need  to overcome multiple sources of unwanted heat through the regions  before the system can cool MIRI. Specifically, the cooling components  span temperatures ranging from approximately 300K (about 80 degrees  Fahrenheit, or room temperature) where the spacecraft is located on the  sun-facing surface of the telescope to approximately 40K (about -388  degrees Fahrenheit) within the ISIM.
section_title: History of Cost Growth, Low Project Reserves, and Schedule Delays
paragraphs: Since entering development in 1999, JWST has experienced significant  schedule delays and increases to project costs. Prior to being approved  for development, cost estimates of the project originally ranged from $1  billion to $3.5 billion with expected launch dates ranging from 2007 to  2011. In March 2005, NASA increased the JWST’s life-cycle cost  estimate to $4.5 billion and delayed the launch date to 2013. We reported  in 2006 that the cost growth was due to a delay in launch vehicle  selection, budget limitations in fiscal years 2006 and 2007, requirements  changes, and an increase in the project’s reserve funding—funding used  to mitigate issues that arise but which were previously unknown. In April  2006, an Independent Review Team confirmed that the project’s technical  content was complete and sound, but expressed concern over the  project’s reserve funding, reporting that it was too low and phased in too  late in the development lifecycle. The review team reported that for a  project as complex as JWST, a 25 to 30 percent total reserve funding was  appropriate. The team cautioned that low reserve funding compromised  the project’s ability to resolve issues, address risk areas, and  accommodate unknown problems. The project was baselined in April  2009 with a life-cycle cost estimate of $4.964 billion—including additional  cost reserves—and a launch date in June 2014. Shortly after JWST was  approved for development and its cost and schedule estimates were  baselined, project costs continued to increase and the schedule was  extended.
In response to a request from the Chair of the Senate Subcommittee on  Commerce, Justice, Science, and Related Agencies to the NASA  Administrator for an independent review of JWST—stemming from the  project’s cost increases and reports that the June 2014 launch date was  in jeopardy—NASA commissioned the Independent Comprehensive  Review Panel (ICRP). In October 2010, the ICRP issued its report and  cited several reasons for the project’s problems including management,  budgeting, oversight, governance and accountability, and communication  issues. The panel concluded JWST was executing well from a technical  standpoint, but that the baseline funding did not reflect the most probable  cost with adequate reserves in each year of project execution, resulting in  an unexecutable project. Following this review, the JWST program  underwent a replan in September 2011 and was reauthorized by  Congress in November 2011, which placed an $8 billion cap on the  formulation and development costs for the project. On the basis of the  replan, NASA announced that the project would be rebaselined with a life- cycle cost at $8.835 billion—a 78 percent increase—and would launch in  October 2018—a delay of 52 months. The revised life-cycle cost estimate  included 13 months of funded schedule reserve. In the President’s Fiscal  Year 2013 budget request, NASA reported a 66 percent joint cost and  schedule confidence level associated with these estimates. A joint cost  and schedule confidence level, or JCL, is the process NASA uses to  assign a percentage to the probable success of meeting cost and  schedule targets and is part of the project’s estimating process. Figure 3  shows the original baseline schedule and the revised 2011 baseline for  JWST.
As part of the replan in 2011, JWST was restructured and is now a single  project program reporting directly to the NASA Associate Administrator for  programmatic oversight and to the Associate Administrator for the  Science Mission Directorate for technical and analysis support. Goddard  Space Flight Center is the NASA center responsible for the management  of JWST. See figure 4 for the current JWST organizational chart.
section_title: GAO Reviewed JWST Project in 2012
paragraphs: In 2012, we reported on numerous technical challenges and risks the  project was facing. For example, a combination of numerous instrument  delays and leaks in the cryocooler’s bypass valves resulted in the use of  18 of ISIM’s 26 months of schedule reserve and the potential for more  schedule reserve to be consumed. Additionally, we identified that the  current JWST schedule reserve lacked flexibility for the last three  integration and testing events (OTIS, the spacecraft, and observatory),  planned for April 2016 through May 2018. While there was a total of 14  months of schedule reserve for all five integration and test events—when  problems are more likely to be found—only 7 months were likely to be  available for these last three efforts. We also reported that the spacecraft  exceeded the mass limit for its launch vehicle and that project officials  were concerned about the mass of JWST since the inception of the  project because of the telescope’s size and limits of the launch vehicle.  In addition to these technical challenges, we reported that the lack of  detail in the summary schedule used for JWST’s JCL analysis during the  2011 replan prevented us from sufficiently understanding how risks were  incorporated, calling into question the results of that analysis and,  therefore, the reliability of the replanned cost estimate.
In our December 2012 report, we made numerous recommendations  focused on providing high-fidelity cost information for monitoring project  progress and ensuring technical risks and challenges were being  effectively managed and sustaining oversight. One recommendation was  that the project should perform an updated integrated cost/schedule risk,  or JCL, analysis. In addition, we recommended that the JWST project  conduct a separate review to determine the readiness to conduct  integration and test activities prior to the beginning of the OTIS and  spacecraft integration and test efforts. NASA concurred with these two  recommendations.
section_title: JWST Project Is Generally Executing to Its 2011 Cost and Schedule Baseline; Recent Performance Has Declined
paragraphs: The JWST project is generally executing to its September 2011 revised  cost and schedule baseline. Through the administration’s annual budget  submissions, NASA has requested funding for JWST that is in line with  the rebaseline plan and the project is maintaining 14 months of schedule  reserve to its October 2018 launch date. Cumulative performance data  from the prime contractor, which is responsible for more than 40 percent  of JWST’s remaining $2.76 billion in development costs, indicate that  work is being accomplished on schedule and at the cost expected.  Monthly cost and schedule metrics, however, indicate that this  performance has been declining since early 2013. The JWST project is  maintaining oversight established as part of the replan, for example, by  continuing quarterly NASA and contractor management meetings and  instituting a cost and schedule tracking tool for internal efforts. The  project, however, is not planning to perform an updated integrated cost  and schedule risk analysis, which would provide management and  stakeholders with information to continually gauge progress against the  baseline estimates.
subsections: section_title: Rebaselined Cost Commitments Are Being Met; Recent Work Is Costing More Than Planned
paragraphs: The JWST project is executing to the cost commitment agreed to during  the September 2011 rebaseline. Since that time, NASA’s funding  requests for JWST have been consistent with the budget profile of the  new cost rebaseline. For fiscal year 2013, the funding the project  received—almost $628 million—matched the agency’s budget request. In  addition, the project has been able to absorb cost increases on various  subsystems through the use of its cost reserves. Project officials remain  confident that they can meet their commitments, and stay within an $8  billion development cost cap recommended by congressional conferees,  if funding is provided as agreed during the replan.
Performance data from contractors show that planned work was generally  being performed within expected costs, but performance has declined  over the past year. The project collects earned value management (EVM)  cost data from several of its major contractors and subcontractors. EVM  data for Northrop Grumman—the project’s prime contractor which is  responsible for more than 40 percent of the remaining development  costs—indicates that cumulatively from May 2011 work planned is being  performed at the cost expected. This measure, known as the cumulative  cost performance index (CPI), provides an indication of how a contractor  has performed over an extended period of time. The CPI indicates that  until June 2013 the contractor performed slightly more work for the cost  incurred than what was expected. Recent monthly performance,  however, has begun to lower the cumulative index. From December 2012  until June 2013, monthly CPI data, which gives an indication of current  performance, show that the contractor has been accomplishing less work  than planned for the cost incurred. See figure 5.
Although several subsystems are experiencing positive performance, cost  overruns on spacecraft-related development activities are contributing to  this recent trend. For example, Northrop Grumman has reported negative  performance within the spacecraft systems engineering and the electrical  power subsystems activities for a 6-month period as of the end of June  2013. We calculate that this contract, which is approximately two-thirds  complete, could experience a slight cost overrun based on current data.  Northrop Grumman is using cost management reserves to offset the  decline in performance, but the JWST project reports that Northrop  Grumman is consuming cost reserves at a rate faster than planned.
Contractor EVM cost data for ITT/Exelis—which is providing services  related to the OTE and OTIS integration and test efforts—also indicate  that in recent months the contractor has been accomplishing less work  than planned for the cost incurred. ITT/Exelis has experienced cost  overruns in each month from March through June 2013, which has  lowered the cumulative CPI to 0.98. Project officials told us that the  ITT/Exelis has sufficient cost reserves to offset the recent cost overruns  and that a cumulative CPI of 0.98 is within the range of acceptable  performance. Best practices indicate that a CPI of 1.0 or above is  favorable. We found small cost overruns across many elements of the  work being performed by ITT/Exelis, similar to the analysis performed by  the project. Based on our analysis of EVM data through the end of July  2013, we estimate that this contract could experience a small cost  overrun. As of July 2013, ITT/Exelis had completed a little more than one- third of the planned work for this contract and used more than 44 percent  of available management reserves from October 2012 to July 2013.
In addition to the work being performed by contractors, the JWST project  also performs development work internally at NASA’s Goddard Space  Flight Center. For example, the project internally manages the ISIM  development effort that is expected to cost over $1 billion, which includes  the first of five major integration and test efforts. The current estimated  cost at completion for ISIM as calculated by the project has risen more  than $109 million—a 9.8 percent increase—since the 2011 rebaseline of  the project. The cost overrun is primarily because of late instrument  deliveries and is being accommodated through the use of project  reserves.
section_title: JWST Is Executing to Its Rebaselined Schedule; Recent Work Is Taking Longer Than Planned
paragraphs: The JWST project is executing to the baseline schedule commitment  agreed to during the September 2011 rebaseline. The JWST project  continues to report 14 months of schedule reserve to its October 2018  launch date, pending a review of the need to use schedule reserve based  on the impacts of the government shutdown in October 2013. See figure  6.
We found in 2012 that the 7 months of schedule reserve held by the OTE  subsystem will likely be used during its integration and test, prior to  delivery to OTIS. If the OTE integration and test effort uses schedule  reserve beyond those 7 months, it will reduce the amount of schedule  reserve available for the last three integration and test efforts. Northrop  Grumman officials said that the OTE integration and test effort is very  sequential and does not offer much flexibility to allow for changes to the  process flow. The integration and test of OTE must be complete for the  OTIS integration effort to begin on schedule. In December 2013, the  project indicated that the 14 months of total schedule reserve held by the  project was being assessed due to delivery problems with portions of the  observatory’s sunshield and the impact of the government shutdown.
Because of instrument and hardware delays and non-availability of a test  chamber, the project now reports 7 months of schedule reserve  associated with the ISIM integration and test effort before it is needed for  integration with the OTE subsystem to form OTIS. Previously, the project  reported that ISIM had almost 8 months of schedule reserve, which did  not account for the delayed start of the first scheduled cryo-vacuum test— in which a test chamber is used to simulate the near-absolute zero  temperatures in space. The current 7 months of schedule reserve for the  ISIM integrations and test effort does not include the impact of any  potential delays due to the government shutdown in October 2013, which  was still being determined in mid-December 2013. The first cryo-vacuum  test was considered a risk reduction test by the project because it did not  include two of the project’s four instruments and was to test procedures  and the ground support equipment to be used in later cryo-vacuum tests  of ISIM. During the replan, this test was scheduled to begin in February  2013, but was delayed until August 2013 because of several issues,  including availability of the test chamber and delays in development and  delivery of a radiator for the harness that holds electrical wiring. Project  officials said they will adjust the ISIM schedule to minimize the schedule  impact by performing some activities concurrently, delaying some  activities until after the first cryo-vacuum test, and removing some  activities. They added that a recently approved September 2013 revision  to the ISIM schedule only reduced schedule reserve by 1 week and no  additional risk will be incurred based on these changes to the ISIM  schedule. The two subsequent cryo-vacuum tests, however, have slipped  up to 2 months in the latest revision to the ISIM schedule, although  project officials state that the April 2016 completion date for ISIM testing  and delivery to the OTIS integration and test effort remains unchanged.  According to the JWST program manager, however, the first cryo-vacuum  test was in process when the government shutdown happened and,  although many of the testing goals were accomplished through  prioritization of test activities, the test was terminated once the ISIM staff  resumed work and some activities were not accomplished. As a result, he  said that the project would incur more risk in the second cryo-vacuum test  that is currently scheduled to start in April 2014.
In addition to maintaining up to 14 months of schedule reserve, the  project is generally meeting the milestones it reports to Congress and  other external entities. See table 1. These milestones include technical  reviews prior to the spacecraft critical design review, hardware tests, and  the delivery of key pieces of hardware. As shown in the table, the project  has completed the majority of its milestones as planned and has deferred  six milestones in the past 2 fiscal years. Among the deferred milestones  are delays to completion of the first ISIM cryo-vacuum test and delivery of  flight hardware for the MIRI instrument cryocooler.
EVM schedule data for Northrop Grumman indicates that the cumulative  planned work since the new schedule estimate was agreed upon is being  performed as expected. This measure, known as the cumulative schedule  performance index (SPI), shows consistent performance at the aggregate  level for the past year. However, monthly SPI metrics indicate a slight  decline in performance in 9 of the 12 months between August 2012 and  July 2013. See figure 7.
The data from Northrop Grumman in recent months indicates that work is  slightly behind schedule for the spacecraft subsystem.
section_title: JWST Project Maintained and Enhanced Oversight Initiatives, but is Not Planning to Perform Updated Cost and Schedule Risk Analysis to Support Monitoring of Project Progress
paragraphs: The JWST project has maintained the oversight activities put in place  following the replan and added additional oversight mechanisms. For  example, some of the oversight activities implemented as part of the 2011  replan that are still ongoing include
The JWST Program Director is holding monthly meetings with the
The JWST Program Director is holding quarterly meetings with  Northrop Grumman senior management and the Goddard Space  Flight Center Director, and
The JWST Project Spacecraft Manager has relocated to provide an  on-site presence at the Northrop Grumman facility.
The project also has implemented some new oversight mechanisms since  the time of our last review in 2012, according to JWST officials. For  example, the project is implementing a tool to continually update the cost  estimate for the internal work on the ISIM development activities. In  addition, the project is working with the Space Telescope Science  Institute to design a tool, similar to EVM, to monitor progress on ground  systems development. The project also has added a financial analyst at  the Northrop Grumman facility to provide the spacecraft manager and the  project ongoing and increased financial insight of the work being  performed by Northrop Grumman and to analyze monthly data prior to the  monthly project business meetings with the contractor. In response to our  prior recommendation, the project has modified its schedule to add an  independent review prior to the beginning of the OTIS and spacecraft  integration and test efforts.
Despite these improvements in oversight, JWST project officials said that  they are not planning to perform an updated integrated cost/schedule risk  analysis—or joint cost and schedule confidence level (JCL) analysis as   GAO’s cost estimating best practices call for  we recommended in 2012. a risk analysis and risk simulation exercise—like the JCL analysis—to be  conducted periodically through the life of a program, as risks can  materialize or change throughout the life of a project.updated on a regular basis, the cost estimate cannot provide decision  makers and stakeholders with accurate information to assess the current  status of the project. As we recommended in 2012, updating the project’s  JCL would provide high-fidelity cost information for monitoring project  progress. While NASA concurred with our recommendation, project   Unless properly  officials have subsequently stated that they do not plan to conduct an  updated JCL. A program official stated that the project performs monthly  integrated programmatic and cost/schedule risk analyses using various  tools and that the information that these tools provide is adequate for their  needs. For example, the JWST project conducts on-going risk  identification, assigning probability and dollar values to the risks, tracks  actual costs against planned costs to assess the viability of current  estimates, uses earned value management, and performs schedule  analyses.
Moreover, while the JWST program manager acknowledged that NASA  concurred with our recommendation, he said that the agency interpreted  that it would be sufficient to do these lower level analyses instead of  performing an updated JCL. NASA, however, has not addressed the  shortcomings of the schedule that supports the baseline itself. For  example, we found that the lack of detail in the summary schedule used  for JWST’s last JCL in May 2011 prevented us from sufficiently  understanding how risks were incorporated, therefore calling into question  the results of that analysis. Since the JCL was a key input to the decision  process of approving the project’s new cost and schedule baseline  estimates, we maintain that the JWST project should perform an updated  JCL analysis using a schedule that should now be much more refined and  accurate and has sufficient detail to map risks to activities and costs in  addition to the other analyses they currently perform. Doing so could help  increase the reliability of the cost estimate and the confidence level of the  JCL. Furthermore, risk management is a continuous process that  constantly monitors a project’s health. The JWST project is still executing  to a plan that was based on the JCL performed in May 2011. The risks  the project is currently facing are different than those identified during the  JCL process more than 2 years ago, and will likely continue to evolve as  JWST is still many years from launch.
section_title: Project Made Progress Addressing Technical Risks; Challenges Persist
paragraphs: The JWST project has made progress in addressing some technical risks;  however, other technical challenges exist that have caused development  delays and cost increases at the subsystem level. The project and its  contractors have nearly addressed a problematic valve issue in the MIRI  cryocooler that has been a concern for several years, the OTE and ISIM  development efforts have made progress over the past year, and both the  project and contractors have remedied the spacecraft mass issue that we  reported on last year. The project has other technical issues, however,  that still need to be resolved. For example, there is a separate and  significant performance issue with the cryocooler and though project  officials state that they understand the issue, the subcontractor is still  working to validate the changes made to the cryocooler to address the  issue. These issues with the cryocooler have led to an increase of about  120 percent in cryocooler contract costs and the execution of the  remaining cryocooler effort will be challenging. In addition, the OTE and  ISIM efforts are still addressing risks that threaten their schedules.
subsections: section_title: Despite Progress, Cryocooler Performance Issues Led to Multiple Replans, Increased Costs, and Schedule Delays
paragraphs: Despite progress in some areas, the cryocooler development effort has  been and remains a technical challenge for the project. The cryocooler  subcontractor has addressed much of the valve leak issue that we  reported on in 2012, and all but the last of the replacement valves, which  were produced with new seal materials, have successfully completed  testing. While resolution of this issue will be a positive step for the project,  other, still unresolved issues with the cryocooler have arisen that have  required additional cost and schedule resources to address. Specifically,  a key component of the cryocooler underperformed prior tests of this  technology by about 30 percent. In addition, both the Jet Propulsion  Laboratory (JPL)—which awarded the cryocooler subcontract—and the  subcontractor were focused on addressing the valve issue, which limited  their attention to the cooling underperformance issue. In late 2012, the  cryocooler subcontractor reported that it would be unable to meet the  cryocooler schedule. The subcontractor is working toward a revised test  schedule, agreed upon in April 2013, which delays acceptance testing  and includes concurrent testing of hardware. In August 2013, the  cryocooler subcontract was modified to reflect a 69 percent cost increase.  Additionally, the number of subcontractor staff assigned to the cryocooler  subcontract has increased from 40 to approximately 110, which accounts  for a significant portion of the cost increase. This was the second time in  less than 2 years that the cryocooler subcontract was modified.  Cumulatively, the cryocooler subcontract value has increased by about  120 percent from March 2012.
Various issues may have contributed to the current problems with the  cryocooler. For example, according to project and JPL officials they had  not verified the cryocooler cost and schedule estimates provided by the  subcontractor prior to the project establishing new baseline cost and  schedule estimates in 2011. Doing so may have allowed them to ensure  adequate resources were accounted for in the new baseline estimates.  JPL officials stated that the subcontractor proposal was verified prior to  the completion of the March 2012 cryocooler replan. The subcontractor,  however, reported that the 2012 replan did not include cost or schedule  allowance for rework should additional problems arise, which did happen.  In addition, despite erratic and negative EVM data from the subcontractor  immediately following the March 2012 cryocooler replan, an in-depth  review was not initiated until 9 months later by the cryocooler  subcontractor. JPL officials stated that, during this time, they were  performing analysis of the EVM data and the technical progress of the  subcontractor and provided the results of their analysis to the project.  Finally, the project had not followed key best practices since early in  development, which left it at an increased risk of cost and schedule  delays. For example, best practices call for testing of a model or  prototype of a critical technology in its flight-like form, fit, and function and  in a simulated realistic or high fidelity lab environment by its preliminary  design review. While the subcontractor tested a demonstration model of  the cryocooler in such an environment and the project assessed the  technology as mature in 2008, a project official acknowledges that the  demonstration model’s mechanical design was different than what would  be used in space and, according to that official, those differences led to  the loss of performance between the demonstration model and the  current cryocooler. In addition, only 60 percent of the cryocooler’s  expected design drawings were released as of the mission critical design  review—well below the best practice standard of 90 percent drawings  released by critical design review—indicating that the project moved  forward without a stable cryocooler design as well as an immature  cryocooler technology, which increases risk.
The execution of the remaining cryocooler schedule will continue to be  challenging as the performance issue is not resolved, the revised  schedule is optimistic, the subcontractor has identified significant risks not  incorporated in the rebaseline, and there are risks associated with the  revised testing approach. The cryocooler subcontractor has developed a  separate verification model, which is now being used to validate that the  cryocooler redesign will address the underperformance. This step is  important because, according to the cryocooler subcontractor program  manager, the internal structures of the cryocooler component are intricate  and once a unit is completed the internal structure cannot be modified.  Thus, when issues arise, such as use of incorrect parts or unexpected  underperformance, a new unit must be built rather than simply changing  parts on the underperforming cryocooler component. Testing of the  verification model, which will give an indication of whether the  performance issue has been rectified and a new flight model can be built,  was scheduled to be complete in October 2013, but has been delayed.  The subcontractor project manager reports that issues were found with  processes used to assemble the verification model that must be resolved  before testing resumes, which is not expected until at least late December  2013. This delay may reduce the amount of schedule margin available to  the overall cryocooler effort.
The cryocooler schedule—agreed upon in April 2013—was optimistic,  according to the cryocooler subcontractor program manager. Shortly after  the new schedule was put in place, he told us that he had low confidence  that the subcontractor would be able to meet this schedule based on the  development issues mentioned above. In addition, the JPL scheduler for  the cryocooler said that he had only moderate confidence of the  subcontractor’s ability to meet this schedule. In line with their concerns,  the cryocooler subcontractor recently depleted all of its schedule reserve  for deliveries to JPL prior to the start of acceptance testing. The  cryocooler subcontractor also identified other risks that could impact its  execution of the subcontract, but that were not included as part of the  rebaseline plan in the modified subcontract. The project retained financial  responsibility for addressing those risks, should they arise, at the project  level by identifying over $8 million in cost reserves in fiscal years 2014  and 2015. However, some of these risks could require significantly more  than $8 million to address. For example, the cryocooler subcontractor  program manager stated that some of these risks, if realized, could take a  year to mitigate. As of September 2013, delivery dates agreed to in April  2013 for all of the major flight and spare cryocooler components have  been delayed, all six weeks of schedule reserve being held at the  cryocooler subcontractor had been exhausted, and the start of  acceptance testing at JPL has been delayed. Any further delays will have  to be accommodated through the use of 12 weeks of schedule reserve  held by JPL. The cryocooler subcontractor also recently began reporting  EVM data based on the latest cost and schedule estimates and, in line  with the delays mentioned above, this data already shows that work is  costing more and taking longer than planned. JPL’s schedule reserve  also has to support any issues that arise during acceptance and end-to- end testing of the cryocooler hardware prior to delivery to the spacecraft  integration and test effort.
In an effort to reduce this risk, the project reordered the integration and  test schedule. This removed some, but not all, of the cryocooler  component testing schedule risk, which may limit the project’s ability to  address issues that arise during component testing. Specifically, two  major spare components of the cryocooler will still be in acceptance  testing when spacecraft integration and test begins in April 2016, which is  also a risk to the spacecraft integration and test schedule. For example, if  a particular cryocooler component fails during one test and a spare  component is still undergoing acceptance testing, then the test schedule  may be delayed waiting for repairs to be made to the component or for  the spare component to be available.
section_title: OTE and ISIM Have Made Progress but Face Risks That May Impact Schedule
paragraphs: Northrop Grumman has made progress on the OTE, but the project  expects the contractor to use its current schedule reserve and the OTE is  facing risks that may impact the schedule if they are realized. Progress  has been made over the past year in fabricating the OTE support  structure, which holds the mirrors and ISIM and connects all the pieces of  the observatory. Specifically, all of the support structure sections have  been completed and fully integrated and the structure has entered  cryovacuum testing. The project is tracking an issue with release  mechanisms holding the spacecraft and the OTE together while stowed  within the launch vehicle and used during the deployment of the  telescope after launch. Currently the mechanisms are causing excessive  shock vibration when released. According to a NASA official, the project  and the contractor are evaluating potential solutions which include  changes to the design of the release mechanism, using damping  materials to lessen the impact to the spacecraft, and testing to see if the  shock requirement can be relaxed. The project has delayed the release  mechanism design review until January 2014—after the spacecraft critical  design review—while it works to mitigate the issue with contractors.  Project officials stated the results of this component level design review  will be evaluated prior to a larger mission review to be held later in 2014.  In December 2013, the project was also assessing the possibility that  portions of the observatory’s sunshield may be delivered up to 3 months  late, which could impact the amount of schedule reserve being held by  the project. The project indicates that it is considering options by the  contractor to recover some of that potential schedule delay.
The project has made progress on various portions of the ISIM as well.  For example, two of the four instruments have been integrated into the  ISIM for testing and fabrication of replacement near infrared detectors  used in three of the four instruments—which we reported in 2012 may  need to be replaced—is ahead of schedule. Prior schedule conflicts with  another NASA project, however, delayed the start of the ISIM integration  and test effort and instrument and component delays are further  threatening the ISIM integration and test schedule which may lead to  additional cost increases. The project has already replanned the ISIM  schedule flow due, in part, to delays with the Near-Infrared Camera  (NIRCam) and Near-Infrared Spectrograph (NIRSpec) instruments.  Specifically, the NIRSpec instrument and NIRCam’s optics were delivered  more than a year behind schedule. NIRSpec completed environmental  testing and was delivered to Goddard in late September 2013. An  electronics component of the NIRCam instrument, however, failed  functional testing following a vibration test possibly due to manufacturing  defects. The contractor has developed an approach to screen similar  components to verify whether those components have similar anomalies.  If the components pass the screening process, then environmental testing  will continue with a spare in place of the component that malfunctioned. If  all of the components show similar anomalies, they will be restricted from  vibration tests and used in other testing until replacement components  are ready. This issue may impact the already delayed start of the second  and third ISIM cryo-vacuum tests, which would further compress the ISIM  integration and test schedule or require the project to use some of ISIM’s  schedule reserve. Because the ISIM schedule has already been  compressed, the project will have less flexibility should any issues or  delays arise during this effort. The project is covering the current ISIM- related cost increase—9.8 percent—primarily with funding reserves.  Extending the length of time needed to conduct the ISIM integration and  test effort, should there be further delays, would require maintaining test  personnel and facilities longer than planned, which may lead to further  cost increases.
section_title: Northrop Grumman Has Addressed Spacecraft Mass Issues
paragraphs: Northrop Grumman has successfully addressed the spacecraft mass  issue that we reported on in 2012 and project officials state that they are  comfortable with the observatory mass margin as the project heads into  multiple major integration and test efforts, despite the mass margin being  lower than Goddard standards.  In December 2012, we reported that the  spacecraft was more than 200 kilograms over its mass allocation. In  November 2013, Northrop Grumman officials stated that the spacecraft  was under its mass allocation at that time. Since December 2011, both  the contractor and the project made mass reduction a priority and the  contractor currently has margin available to address future issues that  may require additional mass to solve. The project’s current overall mass  margin is approximately 7.7 percent, which does not include 90 kilograms  of additional mass allocation the project received in 2013 from the launch  vehicle provider. This is lower than the Goddard standard of 15 percent  mass margin at this phase of development. According to project officials,  they applied the Goddard standard at the subsystem level rather than at  the observatory level due to JWST’s complexity, which allowed them to  maintain a lower overall observatory mass margin. They added that the  observatory and its component elements have an acceptable amount of  mass margin as the project enters its major integration and test efforts  and, while they will maintain standard mass controls to avoid  unnecessary growth, they do not expect mass margins to be a significant  concern going forward. We plan to continue to monitor mass margin in  future reviews as the project proceeds through integration and test efforts.
section_title: Near-term Funding Constraints May Limit Project’s Ability to Meet Future Commitments
paragraphs: Several current near-term funding constraints such as low cost reserves,  a higher-than-expected rate of spending, and potential sequestration  impacts are putting at risk NASA’s ability to meet its cost and schedule  commitments for JWST. In September 2013, project officials reported that  while they are making good technical progress, the level of cost reserves  held by the project in fiscal year 2014 had become the top issue facing  the project and may require them to defer future work. Although not  currently identified as an issue by the project, a significant portion of fiscal  year 2015 project-held cost reserves have also already been allocated.  This does not take into account reserves held by the JWST program at  NASA headquarters in fiscal years 2014 and 2015 that can be used to  supplement reserves held by the project. However, fiscal year 2014  program reserves are minimal compared to future years. As of September  2013, the project has allocated approximately 60 and 42 percent of its  reserves in fiscal years 2014 and 2015, respectively. See figure 8.
The need to allocate a significant portion of cost reserves in fiscal year  2014 and 2015 has been driven primarily by the technical issues with the  MIRI cryocooler. Specifically, the subcontract modification resulting from  the cryocooler replan required the allocation of over $25 million of cost  reserves in fiscal year 2014 and 2015. After allocation of these cost  reserves, the project began tracking the risk of low fiscal year 2014 cost  reserves.
Project officials report that the project’s low reserve posture in fiscal year  2014 may require them to defer work to future years. Specifically,  because the project continues to maintain 14 months of funded schedule  reserve, it may begin using some of that schedule reserve to conduct  work later or allow work to take longer than planned. There are risks  associated with this approach, however. For example, prior to the  project’s replan in 2011, low cost reserves and technical challenges  forced project management to defer planned work into future years. This  ultimately led to increased costs for the deferred work and a schedule that  was unsustainable. Much of the remaining work on JWST involves the  five major integration and test efforts—which began in fiscal year 2011— during which work is often sequential in nature and cost and schedule  growth typically occurs. Depleting schedule reserve now could impact  project officials’ ability to address technical risks or challenges not  currently identified or realized, but that will likely arise during this phase.  Project officials said that they would like to strike a balance between  using remaining cost reserves and having to utilize schedule margin to  complete planned work and address currently unknown technical  challenges, but their goal is to use as little schedule margin as possible in  fiscal year 2014.
Northrop Grumman has also identified issues with the adequacy of its  cost management reserves in fiscal year 2014. The project shares this  concern given that Northrop Grumman’s cost reserves are eroding faster  than anticipated. As of October 2012, the contractor held more than $244  million in cost management reserves for the remainder of the contract, but  has used almost 24 percent of those management reserves since then.  The approximately $185 million in cost management reserves Northrop  Grumman has available as of September 2013 represents the total  amount of reserves available through the remainder of the contract— almost 6 years—and not how much is available for use specifically for  fiscal year 2014. The contract modification for the 2011 replan was signed  in December 2013 and, according to the Northrop Grumman program  manager, the amount of management reserve available will likely  increase by more than $45 million once budget distributions are  completed by the end of January 2014. In June 2013, Northrop Grumman  had identified up to $80 million in potential risks for fiscal year 2014.  Project officials said that Northrop Grumman will sometimes fund new  contract requirements for future fiscal years with current year cost  reserves. These officials added that they are in the process of  determining whether the rate Northrop Grumman is spending cost  reserves is a result of additional requirements or because of performance  issues. According to JWST project analysts, Northrop Grumman cost  management reserves also remain a challenge in fiscal year 2015 when  compared to the potential threats. The JWST project manager said that  the project could rephase some planned Northrop Grumman cost  management reserves from future years to fiscal year 2014 instead, but  that would require the project to use some of its fiscal year 2014 cost  reserves, which as noted are already constrained. As noted earlier, the  JWST Program at NASA headquarters maintains another set of cost  reserves that could be used to help in situations such as this, but the bulk  of these reserves will not be available until fiscal year 2015.
The project’s rate of spending in fiscal year 2013 could also be a  significant issue if it continues into fiscal year 2014 and officials have  begun tracking the rate of spending as a risk. The project spent  approximately $40 million more than planned in fiscal year 2013.  According to program officials, the amount of this overage is becoming  significant not because of a lack of funds in fiscal year 2013, but because  the fiscal year 2014 budget and project cost reserves are constrained.  Project officials said that they planned to carry over funding from fiscal  year 2013 to support approximately 2½ months of work to help fund  contracts and ensure continued operations during a potential continuing  resolution or other periods of funding uncertainty. If the project were to  receive its full funding allocation for fiscal year 2014 at the level planned,  this 2013 money would supplement the money available to the project in  2014. But if the current rate of spending is sustained, the project would  only carry over enough 2013 money to fund the project for about 7 to 8  weeks into fiscal year 2014. The lower amount of funding carried over will  also cause the project to have less available to supplement shortfalls in  future years. For example, the JWST program manager told us that  Northrop Grumman has requested more funding in fiscal year 2014 than  the amount planned. Program officials noted that if the project continues  to spend in fiscal year 2014 at a rate experienced during the latter part of  fiscal year 2013, it may not be able to carry any funds into fiscal year  2015 as planned. Project officials, however, indicate that they are  confident that they will carryover funds into fiscal year 2015. Our review of  the data found that the project’s increased spend rate in fiscal year 2013  is due mainly to additional resources necessary for the ISIM due to late  hardware deliveries, the cryocooler effort, and the Northrop Grumman  effort to prepare for the spacecraft critical design review in January 2014.
NASA’s ability to remedy these issues will likely be significantly hindered  by the potential impacts from sequestration and competing demands from  other major projects. For example, while NASA officials report that the  agency was able to absorb the sequestration-related reductions in fiscal  year 2013 with relatively no impact on its major projects, including JWST,  they indicate that the agency cannot sustain all of its long-term funding  commitments at sequester levels in fiscal year 2014 and beyond.  Importantly, the JWST project recently began tracking a risk for the  budget uncertainty due to sequestration. The risk outlines that there is a  potential cut to the JWST budget starting in fiscal year 2014, which could  adversely affect the execution of the project’s current plan and potentially  jeopardize the October 2018 launch date. The program office indicates  that NASA headquarters directed JWST to plan for its fiscal year 2014  budget to be consistent with the replan. This direction by NASA could  have an impact on other major NASA projects. In interviews for several  other major NASA projects, officials informed us that they have less than  adequate funding in fiscal year 2014 and some have requested that the  agency rephase funds from later years to fiscal year 2014 to address the  issue. If additional funds are required and prioritized for JWST, there  could be a potentially significant impact on these and other projects within  the agency that are already reporting funding issues in fiscal year 2014.
section_title: Selected JWST Schedules Demonstrate Many Best Practices, but Identified Weaknesses Erode Confidence in the Project’s Integrated Master Schedule
paragraphs: The reliability of the JWST integrated master schedule is questionable  because some of the 23 subordinate schedules synthesized to create it  are lacking in one or more characteristics of a reliable schedule. Schedule  quality weaknesses in the JWST subsystem schedules transfer to the  integrated master schedule. We found a similar result this year consistent  with our analysis in 2012 in which weaknesses in the two subsystem  schedules we analyzed undermined the reliability of the integrated master  schedule.
According to scheduling best practices, the success of a program  depends in part on having an integrated and reliable master schedule that  defines when work will occur, how long it will take, and how each activity  is related to the others that come both before and after it. If the schedule  is dynamic, planned activities within the schedule will be affected by  changes that may occur during a program’s development. For example, if  the date of one activity changes, the dates of its related activities will also  change in response. The master schedule will be able to identify the  consequences of changes and alert managers so they can determine the  best response. The government project management office, in this case  the JWST project office at Goddard Space Flight Center, is ultimately  responsible for the integrated master schedule’s development and  maintenance.
The quality and reliability of three selected subsystem schedules we  examined for this review—ISIM, OTE, and cryocooler—were inconsistent  in following the characteristics of high-quality, reliable schedules. Using  the 10 best practices for schedules, we individually scored and evaluated  the schedules for these subsystems. We then grouped the best practices  into one of four characteristics: comprehensive, well-constructed,  credible, and controlled. The individual best practice scores within each  characteristic were then combined to determine the final score for each  characteristic. See appendix III for more detailed information on each  characteristic and its corresponding best practices. The ISIM and OTE  schedules had more strengths than weaknesses, substantially meeting  three of four characteristics of a reliable schedule. The cryocooler  schedule demonstrated weaknesses in both of the characteristics we  examined. We selected these three subordinate schedules because  they represent the significant portion of ongoing work for the project and  reflect work by the project, the prime contractor, and a subcontractor.  Table 2 identifies the results of each of the selected JWST subordinate  schedules and their corresponding best practices sub scores.
Of the four characteristics of a reliable schedule that we assessed for the  ISIM schedule, we found that three substantially met the criteria— comprehensive, well-constructed, and controlled—while the credible  characteristic was partially met. The strengths of the ISIM schedule were  that it  captured all activities in manageable durations with their proper  sequence,  identified the longest continuous sequence of activities in the  schedule, known as its critical path, and estimated reasonable amounts of total float, defined as the time  activities can slip before delaying key delivery dates.
NASA also maintains a baseline schedule that is regularly analyzed and  updated as progress is made. However, the schedule lacked a schedule  risk assessment—a best practice that gives decision makers confidence  that the estimates are credible based on known risks and allows  management to account for the cost of a schedule slip when developing  the life-cycle cost estimate. Without a schedule risk assessment decision  makers may not obtain accurate cost impacts when schedule changes  occur. Officials noted that while a schedule risk assessment was not  performed on the ISIM schedule itself, the schedule was included as a  part of the overall JWST JCL analysis, and subsequent cost and schedule  estimate, conducted during the project replan in 2011. However, our  analysis of the 2011 JCL indicated that the estimate’s accuracy, and  therefore the confidence level assigned to the estimate, was reduced by  the quality of the summary schedule used for the JCL because it did not  provide enough detail to determine how risks were applied to critical  project activities.
subsections: section_title: Optical Telescope Element
paragraphs: Of the four characteristics of a reliable schedule that we assessed for the  OTE schedule, we found that the comprehensive characteristic was fully  met, credible and controlled characteristics were substantially met, and  the well-constructed characteristic was partially met. The strengths of the  OTE schedule were that it  captured all activities in manageable durations with their proper  sequence,  identified the resources needed for each activity,  linked activities to the final deliverables the work in the schedule is  intended to produce, and accurately reflected dates presented to management in high-level  presentations.
Northrop Grumman, the creator and manager of the schedule, also  maintains a baseline schedule that is regularly analyzed and updated as  progress is recorded by schedule experts. However, while Northrop  Grumman has identified a critical path, our analysis was not able to  confirm that this path described activities in the schedule that were truly  driving the key delivery date for the OTE, which is the delivery of the OTE  to the OTIS testing and integration at Goddard Space Flight Center on  April 28, 2016. Identifying a valid critical path is essential for management  to identify and focus upon activities which will potentially have detrimental  effects on key project milestones and deliverables if they slip. In addition,  we found that one-third of the remaining activities and milestones had  over 200 days of total float. This means that, according to the schedule,  these activities could be delayed 9 working months without impacting the  key delivery date. Realistic float values allow managers to see the impact  of a delayed activity on future work. However, unrealistic estimates of  float make it difficult to know the amount of time one event can slip  without impacting the project finish date. In addition, incorrect float  estimates will result in an invalid critical path. Northrop Grumman officials  agreed with our assessment but noted the high values of total float are  due to their planning process which only details out the schedule in 6  month increments. Activities beyond the detailed planning window of the  schedule have high float and those estimates of float will become more  reasonable as the schedule is planned in detail. However, best practices  state that all activities in the schedule, even far-term planning packages,  should be logically linked in such a way as to portray a complete picture  of the program’s available float and its critical path. Finally, a schedule  risk assessment has not been conducted on the OTE schedule since  2011. Northrop Grumman officials stated that they are not contractually  required to periodically conduct a schedule risk assessment. However, as  with the ISIM, without a schedule risk assessment decision makers may  not have accurate cost impacts when schedule changes occur.
section_title: Cryocooler
paragraphs: Of the two characteristics of a reliable schedule that were assessed for  the cryocooler schedule, the well-constructed and credible characteristics  were both partially met. The strengths of the cryocooler schedule were  that it had a logical sequence of activities with few missing logic links, and  few issues with incorrect logic that might impair the ability of the schedule to forecast dates dynamically.
Despite these strengths, two of the ultimate goals of a reliable schedule— determining a valid critical path and realistic total float—were only partially  achieved. Officials stated that the schedule is used to manage critical  paths to six major hardware deliveries, or key delivery dates. However,  we could not determine how the schedule is used to identify and present  those paths to management. In addition, the use of date constraints in 19  activities within the schedule helps determine the remaining total float to  some deliveries, but causes an overabundance of activities to appear as  critical, which interferes with the identification of the true project-level  critical path. We also found that while the schedule accurately reflected  some of the delays the project is currently experiencing, its schedule  appears to be overly flexible in some cases, such as having activities with  over 500 days—or over 2 working years—of total float. Incorrect float  estimates may result in an invalid critical path and an inaccurate  assessment of project completion dates.
The schedule also lacks a complete and credible schedule risk analysis,  without which managers cannot determine the likelihood of the project’s  completion date, how much total schedule risk reserve funding is needed,  risks most likely to delay the project, or how much reserve funding should  be included for each individual risk. Northrop Grumman officials, who  manage the schedule and the project, stated that a schedule risk analysis  was performed in March 2013, but the results were not used by JPL  management who oversees the contract. The results of the schedule risk  analysis may help JPL determine the probability of meeting key dates or  how much schedule contingency is needed.
Officials provided us examples of the schedule risk analysis output, but  we were not able to confirm their validity because documentation was not  available on the data, risk, or methodologies. In addition to the lack of  documentation, because we found the schedule to be only partially well- constructed, we cannot be sure that the results of the schedule risk  analysis are valid. Given the weaknesses noted above, if the schedule  risk analysis is to be credible, the program must have a quality schedule  that reflects reliable logic and clearly identifies the critical path before a  schedule risk analysis is conducted. If the schedule does not follow best  practices, confidence in the schedule risk analysis results will be lacking.  Without the schedule risk analysis, the project office cannot rely on the  schedule to provide a high level of confidence in meeting the project’s  completion date or identify reserve funding for unplanned problems that  may occur.
section_title: Conclusions
paragraphs: The JWST project has maintained its cost and schedule commitments  since its 2011 replan, has continued to make good technical progress,  and has implemented and enhanced efforts to improve oversight.  Nevertheless, inherent risks continue to make execution of the JWST  project challenging and near-term indicators show that the project is  currently facing challenges that need to be addressed primarily by  increased reserves and progress tracked using the proper tools. Our  report, however, indicates that the project may not have the appropriate  resources and high fidelity information to ensure execution as planned  and provide realistic information to decision makers and other  stakeholders. For example, near-term cost reserves are constrained and  the project is spending at a higher rate than planned. Without adequate  cost reserves in the near-term and if its increased rate of spending  continues, the project may need to defer planned work and delay the  resolution of future and yet unknown threats. These actions could put the  project on a course to repeat past missteps that led to congressional  intervention and the institution of a cap on development costs. In addition,  the effect sequestration would have on available funding for the project in  fiscal year 2014 and beyond is unknown at this point, but could potentially  compound this issue. As a result, NASA may need to make difficult  decisions about funding JWST adequately at the expense of other,  already cash-strapped projects.
Importantly, JWST project officials may not have the necessary  information to determine the impacts of any resource issues because the  project currently lacks a reliable integrated master schedule due to  weaknesses we found in several subschedules. Without a reliable  schedule, project officials cannot accurately manage and forecast the  impacts of changes to the schedule that will likely come about during the  integration and testing periods. Despite these concerns, the JWST project  has declined to take adequate steps to address our recommendation to  perform an updated cost and schedule risk analysis—or JCL—that is  based on current risks and a reliable schedule. Unless properly updated  to include a reliable schedule that incorporates known risks, particularly if  NASA is faced with additional resource constraints through the  continuation of sequestration, the cost estimate will not provide decision  makers with accurate information to assess the current status of the  project.
section_title: Matter for Congressional Consideration
paragraphs: To help ensure that NASA officials are making decisions using up to date  and reliable information about the JWST project, Congress should  consider requiring the NASA Administrator to direct the JWST project to  conduct an updated joint cost and schedule confidence level analysis that  is based on a reliable schedule and current risks.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the NASA Administrator take the following two  actions:  In order to ensure that the JWST project has sufficient available  funding to complete its mission and meet its October 2018 launch  date and reduce project risk, ensure the JWST project has adequate  cost reserves to meet the development needs in each fiscal year,  particularly in fiscal year 2014, and report to Congress on steps it is  taking to do so, and In order to help ensure that the JWST program and project  management has reliable and accurate information that can convey  and forecast the impact of potential issues and manage the impacts of  changes to the integrated master schedule, perform a schedule risk  analysis on OTE, ISIM, and cryocooler schedules, as well as any  other subschedules for which a schedule risk analysis was not  performed. In accordance with schedule best practices, the JWST  project should ensure that the risk analyses are performed on reliable  schedules.
section_title: Agency Comments and Our Evaluation
paragraphs: NASA provided written comments on a draft of this report. These  comments are reprinted in appendix IV.
In responding to a draft of this report, NASA concurred with our two  recommendations; however, in some cases it is either not clear what  actions NASA plans to take or when they will complete the action to  satisfy the intent of the recommendations.
NASA officials concurred with our recommendation to ensure the JWST  project has adequate cost reserves to meet the development needs in  each fiscal year, particularly in fiscal year 2014, and report to Congress  on steps it is taking to do so. In their response, the Acting JWST Program  Director cited NASA and the administration’s request to Congress to  appropriate the full JWST replan level funding for fiscal year 2014, which  includes the level of unallocated future expenses, or cost reserves,  established in the replan. He also commented that NASA conducts  monthly reviews to evaluate risks and associated impacts to funding in  order to ensure that adequate cost reserves are available in each fiscal  year. We acknowledge in our report that the JWST project has been fully  funded at levels commensurate with the 2011 baseline through fiscal year  2013. However, cost reserves approved for the project during the 2011  replan were based on the risks known at that time. The events of fiscal  year 2013 have weakened the project’s financial posture and flexibility the  project has to address any potential technical challenges going forward  into fiscal year 2014 and beyond. In addition, NASA’s response does not  indicate how the agency plans to report to Congress the steps it is taking  to ensure that the JWST project has adequate cost reserves to meet its  October 2018 launch date. We maintain that NASA should provide more  detail to Congress on its plans given the already constrained cost reserve  posture the project has early in fiscal year 2014 and past issues with low  levels of cost reserves that forced the project to defer work, which led to  significant cost increases and schedule delays.
NASA officials concurred with our recommendation to perform a schedule  risk analysis on OTE, ISIM, and cryocooler schedules, as well as any  other subschedules where a schedule risk analysis was not performed  and that, in accordance with schedule best practices, the risk analyses  are performed on reliable schedules. The Acting Program Director stated  NASA will conduct probability schedule risk analyses on the OTE, ISIM,  and cryocooler schedules by the end of calendar 2014 using NASA best  practices. This is a positive step, given that our previous work has found  that GAO and NASA best practices for scheduling are largely  consistent. The Acting Program Director also stated that NASA will  conduct the same analyses for other schedules lacking a risk analysis.  However, no deadline was mentioned for when these analyses will be  accomplished or for how many schedules will be affected. Having reliable  schedules sooner will provide management with more timely and  accurate information on which to make decisions. If the schedule risk  assessments are not completed until after 2014, the project will have less  than 4 years until launch to utilize the information these risk analyses can  provide. Given that we have found reliability issues with the project’s  schedules for the second year, improving the current schedules to meet  best practices is important to provide management with improved tools to  better understand the schedule risks and manage the project.
We are sending copies of the report to NASA’s Administrator and  interested congressional committees. In addition, the report will be  available at no charge on GAO’s website at http://www.gao.gov.
Should you or your staff have any questions on matters discussed in this  report, please contact me at (202) 512-4841 or chaplainc@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to assess (1) the extent to which the James Webb  Space Telescope (JWST) project is meeting its cost and schedule  commitments and maintaining oversight established as part of the  project’s replan, (2) the current major technological challenges facing the  JWST project, (3) the extent to which cost risks exist that may threaten  the project’s ability to execute the project as planned, and (4) the extent  to which the JWST project schedule is reliable based on best practices. In  assessing earned value management (EVM) data from several  contractors and subcontractors and the project’s schedule estimate, we  performed various checks to determine that the data provided was  reliable enough for our purposes.
To assess the extent to which the JWST project is meeting its cost and  schedule commitments and maintaining oversight, we reviewed project  and contractor documentation, analyzed the progress made and any  variances to milestones established during the project’s replan in 2011,  and held interviews with project, contractor, and Defense Contract  Management Agency officials. We reviewed project monthly status  reviews, documentation on project risks, and budget documentation. We  examined and analyzed EVM data from several contractors and  subcontractors. The EVM data reviewed included monthly contractor  performance reports and analysis performed by the JWST project on this  information. For our analysis, we entered only high-level monthly  contractor EVM data into a GAO-developed spreadsheet, which includes  checks to ensure the EVM data provided was reliable enough for our  purposes. We also reviewed the project’s analysis of the estimate at  completion for internal work being performed on the Integrated Science  Instrument Module. We interviewed program and project officials at NASA  headquarters and Goddard Space Flight Center to obtain additional  information on the status of the project with regard to progress toward  baseline commitments. We periodically attended flight program reviews at  NASA headquarters where the current status of the program was briefed  to NASA headquarters officials and members of the Standing Review  Board. We also interviewed JWST project and contractor officials from the  Jet Propulsion Laboratory and Northrop Grumman Aerospace Systems to  determine the extent to which oversight was being conducted. In addition,  we interviewed officials from the Defense Contract Management Agency  to obtain information on oversight activities delegated to it by the JWST  project.
To assess the technological challenges and risks facing the project, we  reviewed project monthly status reviews, information from the project’s  risk database, as well as briefings and schedule documentation provided  by project and contractor officials. These documents included information  on the project’s technological challenges and risks, mitigation plans, and  timelines for addressing these risks and challenges. We also interviewed  program and project officials for each major observatory system to clarify  information and to obtain additional information on system and subsystem  level risks and technological challenges for each subsystem. Further, we  interviewed officials from the Jet Propulsion Laboratory and Northrop  Grumman Aerospace Systems concerning risks and challenges on the  subsystems, instruments, or components they were developing. We  reviewed GAO’s prior work on NASA Large Scale Acquisitions; the  Goddard Space Flight Center Rules for the Design, Development,  Verification, and Operation of Flight Systems technical standards; and  NASA’s Space Flight Program and Project Management Requirements  and Systems Engineering Processes and Requirements policy  documents. We compared Goddard standards with data reported by the  project to assess the extent to which the JWST project followed NASA  policies.
To assess the extent to which cost risks exist that may threaten the  project’s ability to execute the project as planned, we reviewed project  and contractor documentation and held interviews with project and  contractor officials. We reviewed project monthly status reviews and  NASA headquarters flight program reviews, contractor information on the  potential cost to address identified risks, and project analysis of budget- related risks to include the project’s cost reserve posture and the impact  of sequestration. We interviewed program and project officials at NASA  headquarters and Goddard Space Flight Center as well as officials from  the Jet Propulsion Laboratory and Northrop Grumman Aerospace  Systems to obtain information on risks to maintaining cost targets and  plans to mitigate those risks.
To assess the extent to which the JWST project schedule is reliable, we  used GAO’s Schedule Assessment Guide to assess characteristics of  three selected subordinate schedules—the Integrated Science Instrument  Module (ISIM), Optical Telescope Element (OTE), and cryocooler—that  are used as inputs to the integrated master schedule. We selected the  three schedules above as they reflect a significant portion of the work  being conducted within NASA (ISIM), at the contractor level (OTE), and  the subcontractor level (cryocooler) during the course of our work. We  also analyzed schedule metrics as a part of that analysis to highlight  potential areas of strengths and weaknesses against each of our 4  characteristics of a reliable schedule. In order to assess each schedule  against the 4 characteristics and their accompanying 10 best practices,  we traced and verified underlying support and determined whether the  program office or contractor provided sufficient evidence to satisfy the  criterion and assigned a score depicting that the practices were not met,  minimally met, partially met, substantially met, or fully met. By examining  the schedules against our guidance, we conducted a reliability  assessment on each of the schedules and incorporated our findings on  reliability limitations in the analysis of each subordinate schedule. We  also conducted interviews with project and contractor management and  schedulers before our analysis was completed and analyzed project and  contractor documentation concerning scheduling policies and practices.  After conducting our initial analysis, we shared it with the relevant parties  to provide an opportunity for them to comment and identify reasons for  observed shortfalls in schedule management best practices. We took  their comments and any additional information they provided and  incorporated it into the assessments to finalize the scores for each  characteristic and best practice. We were also able to use the results of  the three subordinate schedules to provide insight into the health of the  integrated master schedule since the same strengths and weaknesses of  the subordinate schedules would transfer to the master schedule. We  determined that the schedules were sufficiently reliable for our reporting  purposes and our report notes the instances where reliability concerns  affect the quality of the schedules.
Our work was performed primarily at NASA headquarters in Washington,  D.C. and Goddard Space Flight Center in Greenbelt, Maryland. We also  visited the Jet Propulsion Laboratory in Pasadena, California; Northrop  Grumman Aerospace Systems in Redondo Beach, California; and the  Defense Contract Management Agency in Redondo Beach, California.
We conducted this performance audit from February 2013 to January  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Subsystems of the JWST Observatory
section_title: Appendix III: The Four Characteristics and Ten Best Practices of a High-Quality and Reliable Schedule
subsections: section_title: Schedule characteristic Comprehensive, reflecting
section_title: Well constructed, with
section_title: Controlled, being
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Shelby S. Oakley, Assistant  Director; Karen Richey, Assistant Director; Patrick Breiding; Richard A.  Cederholm; Laura Greifner; Keith Hornbacher; David T. Hulett; Jason  Lee; Sylvia Schatz; Ryan Stott; and Roxanna T. Sun made key  contributions to this report.