James Webb Space Telescope: Project Facing Increased Schedule Risk with Significant Work Remaining
Released: 2014-12-15T00:00:00

Letter

Background

JWST is a large, deployable, infrared-optimized space telescope intended to be the scientific successor to the Hubble Space Telescope. JWST is designed for a 5-year mission to find the first stars and trace the evolution of galaxies from their beginnings to their current formation. It is intended to operate in an orbit approximately 1.5 million kilometers—or 1 million miles—from the Earth. With its 6.5-meter primary mirror, JWST will be able to operate at 100 times the sensitivity of the Hubble Space Telescope. The JWST’s science instruments require extremely cold temperatures, so a tennis-court-sized sunshield will be used to protect the mirrors and instruments from the sun’s heat and allow them to observe very faint infrared sources. The Hubble Space Telescope operates primarily in the visible and ultraviolet regions of the electromagnetic spectrum. The observatory segment of JWST includes several elements (Optical Telescope Element (OTE), Integrated Science Instrument Module (ISIM), and spacecraft) and major subsystems (sunshield, cryocooler).

These elements and major subsystems are being developed through a mixture of NASA, contractor, and international partner efforts. See interactive graphic, figure 1. For more information on the JWST’s organizational structure, see appendix III.

Given JWST’s complexity, it requires a multiyear integration and testing schedule that includes five separate periods over the course of almost 7 years to build the observatory in preparation for launch. See figure 2 for the planned integration and test flow for JWST.

For the majority of the work remaining, the JWST project will rely on three contractors: Northrop Grumman, Exelis, and the Space Telescope Science Institute (STScI). Northrop Grumman plays the largest role, developing the sunshield, the OTE, the spacecraft, and a cooling subsystem for the Mid-Infrared Instrument (MIRI). Northrop Grumman performs most of this work under a prime contract with NASA, but its work on the MIRI cooler is performed under a separate subcontract with the Jet Propulsion Laboratory (JPL). Exelis is manufacturing the test equipment, equipping the test chamber, and assisting in the testing of the optics of JWST. Finally, STScI will collect and evaluate research proposals from the scientific community and will receive and store the scientific data collected by those observations, both of which are services that they currently provide for the Hubble Space Telescope. Additionally, for JWST, STScI will develop and operate the ground system that manages and controls the telescope’s observations on behalf of NASA.

MIRI—one of JWST’s four instruments in the ISIM—requires a dedicated, interdependent two-stage cooler subsystem designed to bring the infrared light detector within MIRI to the required temperature of 6.7 Kelvin (K), just above absolute zero. This major subsystem is referred to as a cryocooler. The cryocooler moves helium gas through 10 meters (approximately 33 feet) of refrigerant lines from the spacecraft located on the sun-facing surface of the JWST observatory to the colder, shaded side where the ISIM is located. According to NASA officials, a cooler of this configuration has never been developed or flown in space before. See figure 3 for a depiction of the cryocooler on JWST.

Project officials stated that the MIRI cryocooler is particularly complex and challenging because of the relatively great distance between the cooling components and the need to overcome multiple sources of unwanted heat through the regions of JWST before the subsystem can cool MIRI’s detector. Specifically, the cooling components span temperatures ranging from approximately 300K (about 80 degrees Fahrenheit, or room temperature) where the spacecraft is located on the sun-facing surface to approximately 40K (about -388 degrees Fahrenheit) within the ISIM.

Cost and Schedule Reserves for NASA Projects

Complex development efforts like JWST must plan to address a myriad of risks and unforeseen technical challenges. To do this, projects reserve extra time in their schedules—which is referred to as schedule reserve— and extra money in their budgets–which is referred to as cost reserve. Schedule reserve is extra time in the project’s overall schedule that is allocated to specific activities, elements, and major subsystems in the event there are delays or to address unforeseen risks. Each JWST element and major subsystem has been allocated schedule reserve. When an element or major subsystem exhausts schedule reserve it may begin to affect schedule reserves on other elements or major subsystems whose progress is dependent on prior work being finished for its activities to proceed. The element or major subsystem with the least amount of schedule reserve determines the critical path for the project. Any delay to an activity that is on the critical path will reduce schedule reserve for the whole project, and could ultimately impact the overall project schedule.

Cost reserves are additional funds that can be used to address unanticipated issues for any element or major subsystem. Cost reserves are used to mitigate issues during the development of a project. For example, cost reserves can be used to buy additional materials to replace a component or, if a project needs to preserve schedule reserve, reserves can be used to accelerate work by adding extra shifts to expedite manufacturing and save time. NASA’s Goddard Space Flight Center (Goddard)—the NASA center with responsibility for managing JWST— has issued requirements that at project confirmation establish both the level of cost and funded schedule reserves that projects must hold. After this point, a specified amount of schedule reserve continues to be required throughout the remainder of development.

History of Cost Growth, Low Project Reserves, and Schedule Delays

Prior to 2011, early technical and management challenges, contractor performance issues, low level cost reserves, and poorly phased funding levels caused JWST to delay work, which contributed to significant cost and schedule overruns. After years of requirements changes, cost growth, low reserve funding, deferred work, and launch delays, the Chair of the Senate Subcommittee on Commerce, Justice, Science, and Related Agencies requested an independent review of JWST. NASA commissioned the Independent Comprehensive Review Panel, which issued its report in October 2010, and concluded that JWST was executing well from a technical standpoint, but that the baseline funding did not reflect the most probable cost with adequate reserves in each year of project execution, resulting in an unexecutable project. Following this review, the JWST program underwent a replan in September 2011 and Congress in November 2011 placed an $8 billion cap on the formulation and development costs for the project. On the basis of the replan, NASA rebaselined JWST with a life-cycle cost estimate of $8.835 billion that included additional money for operations and a planned launch in October 2018. The revised life-cycle cost estimate included a total of 13 months of funded schedule reserve. In the President’s fiscal year 2013 budget request, NASA reported a 66 percent joint cost and schedule confidence level for these cost and schedule baselines. A joint cost and schedule confidence level, or JCL, is the process NASA uses to assign a percentage to the probable success of meeting cost and schedule estimates and is part of the project’s estimating process.

Previous GAO Reviews of JWST Project

In January 2014, GAO found that the JWST project was generally executing to its 2011 revised cost and schedule baselines, but that the most recent work was taking longer than planned. We noted that roughly half of the project’s cost reserves for fiscal years 2014 and 2015 had been committed before those years had even begun, which would constrain JWST’s cost reserve spending in the near term. We reported that while the cryocooler subcontractor was making technical progress, cryocooler development remained an ongoing challenge as there were still unresolved technical issues related to its performance, and its schedule was both optimistic and unreliable. We also found that three of the element and major subsystem schedules the project was using to manage were unreliable and did not meet best practices, which called into question the reliability of the integrated master schedule that is built from those schedules. In 2012, we reported that the accuracy of the project’s cost estimate was lessened because the summary schedule used for the joint cost and schedule confidence level was not detailed enough to determine how risks were applied to critical project activities. We recommended that NASA conduct an updated joint cost and schedule risk analysis to address the issues we identified and use more detailed cost information to adjust its cost estimates. While NASA concurred with our recommendation, officials subsequently stated that they did not plan to conduct an updated joint cost and schedule confidence level analysis, and that the project’s monthly analyses were sufficient for the project’s needs. Thus, we expressed concern in our January 2014 report that NASA’s continued lack of up-to-date cost estimates that incorporate new risks which had emerged since 2011 would inhibit NASA’s ability to effectively monitor its contractors’ progress. We proposed that Congress consider requiring NASA to conduct an updated joint cost and schedule confidence level analysis to address this concern.

Project Risk Heightened Due to Widespread Use of Schedule Reserve to Address Technical Issues

The JWST project has 11 months of schedule reserve remaining—more than required by Goddard standards. In the case of JWST, however, the most significant risks lie ahead in the remaining 4 years, as the project must complete five integration and test periods, three of which have not yet started. Essentially, the risk is defined by bringing all of the complex JWST elements and major subsystems together, testing them, and resolving problems when there is limited schedule left for resolution. All of JWST’s five elements and major subsystems have just weeks of reserve left before their schedules become pacing items on the project’s critical path, potentially reducing the reserve further. More milestones established annually for the various elements and major subsystems have been delayed or deferred during fiscal year 2014 than in the previous 3 years following the replan. For example, the MIRI cryocooler deferred seven milestones until fiscal year 2015 as a result of manufacturing and development delays. Schedule risks are further heightened as the project entered fiscal year 2015 with approximately 40 percent of its cost reserves already committed, leaving fewer dollars available to mitigate other threats to the project schedule.

Flexibility to Address Future Challenges Constrained by Schedule Reserve Loss Prior to Most Integration and Test Efforts

The JWST schedule reserve was diminished this year following the need to address several significant technical challenges across the project’s various elements and major subsystems, and as a result of the October 2013 government shutdown. Specifically, in the past 14 months, the overall project schedule reserve declined from 14 months to 11 months.With less than 4 years until the planned launch in October 2018, the project’s overall schedule reserve is above the Goddard standard and JWST plan—which was set above the Goddard standard and included more reserve than required—for schedule reserve at this point in the project. The scale of JWST’s integration and test effort, however, is more complex than most NASA or Goddard projects. For example, JWST has five major integration and test periods to build the telescope, test the optics, integrate the various elements and major subsystems, and ensure they work properly together. While JWST’s total integration and test cycle runs almost 7 years, the next longest integration and test cycle for a current project managed by Goddard Space Flight Center is over 3 years and the average length of integration and testing for all other current Goddard projects, excluding JWST, is just over 2 years. The JWST project has used over 20 percent of the schedule reserve it held in the past 14 months with almost 4 years remaining for its integration and test effort, where GAO’s prior work has shown and NASA has concurred problems are commonly found and schedules tend to slip.

All individual element and major subsystem schedules have lost schedule reserve since last year. While some use of schedule reserve is expected, this widespread use has resulted in every JWST element and major subsystem schedule being within weeks of becoming the critical path of the project, before most have entered their integration and testing periods. The proximity of all of the element and major subsystem schedules to the critical path means that an issue on any of the schedules may reduce the overall project schedule reserve further, which could then put the overall project schedule at risk. As a result of this, the project has less flexibility to choose which issues to mitigate. As an illustration, the critical path has switched numerous times to several different schedules as a result of using schedule reserve in the past year. See the different amounts of schedule reserve remaining on all elements and major subsystems, their proximity to the critical path and the total schedule reserve for the critical path in figure 4.

Delays were also evident in key milestone slips where more than half of the project-identified key milestones for fiscal year 2014 were completed late or deferred to fiscal year 2015—the most in any year since the replan. Seven of the 11 deferred milestones are related to the development of the cryocooler. The other 4 are due to delays in reviews or assembly for components of the sunshield and spacecraft and late completion of some integration activities for the chamber to test the optics of JWST in 2016. See figure 5 for a list of total milestones and their status.

The project has mitigated a number of issues that could have further impacted the status of schedule reserve by reorganizing work to obtain efficiencies and maintain schedule or stem further losses. For example, a component of the cryocooler had some planned acceptance testing reorganized along with a reduction in the length of time of the testing to accommodate delays. However, with further progression into subsequent integration and test periods, flexibility will be diminished because work during integration and testing tends to be more serial. This is particularly the case with JWST given its complexity.

To some extent, projects have the ability to use cost reserves to maintain schedule. For example, if work is taking longer than expected, a project may be able to fund a second shift of work to speed it up and maintain schedule. In the short term, however, this option may be limited for JWST project officials because approximately 40 percent of the project’s fiscal year 2015 cost reserves were already committed prior to the start of the fiscal year. As such, one of the project’s top issues for fiscal year 2015 is its cost reserve posture, which the project reported is less than desired and will require close monitoring. The project’s cost reserve status for fiscal year 2015 is slightly better than in fiscal year 2014 that we reported on in January 2014, where the project successfully managed low reserves without delaying work. This issue, however, will likely require continued monitoring until fiscal year 2016 when more program level reserves will be available to the project. With schedule losses having already occurred on most elements and major subsystems across the project prior to most of the project’s integration and testing efforts, the fiscal year 2016 funding may arrive too late to prevent further schedule delays and the risk of such delays impacting the launch date is heightened.

Schedule Delays Driven by Significant Design and Manufacturing Challenges with Several Key Elements and Major Subsystems

Cryocooler

The project continues to face major technical challenges building the cryocooler that have significantly delayed delivery of key components, have made it the driver of the project’s overall schedule or the project’s critical path, and required the use of a disproportionate amount of project cost reserves. Since the 2011 replan, the cryocooler development has experienced over 150 percent cost growth. Consequently, it was one of the largest users of project cost reserves in fiscal year 2014 and is contributing to the project’s limited cost reserve status for fiscal year 2015.

All three cryocooler components (the cold head assembly, the compressor assembly, and the electronics assembly) fell significantly behind the delivery dates established during the cryocooler’s April 2013 schedule replan. As we reported in January 2014, the schedule that was adopted during the April 2013 cryocooler replan did not take into account significant risks identified by the cryocooler subcontractor and was overly optimistic—a sentiment that was shared by the cryocooler subcontractor program manager. Moreover, the cryocooler design was not mature at the time of the 2013 replan. The cryocooler subcontractor program manager’s low confidence in the replanned schedule was validated in 2014, when design and manufacturing problems contributed to further delays. In late 2013 and early 2014, implementation of a design change to resolve a manufacturability issue resulted in a 4-month delay to the cryocooler compressor assembly schedule. Additionally, as cryocooler manufacturing efforts accelerated, the cryocooler subcontractor program manager indicated that further delays occurred as a result of design immaturity, manufacturing, and process issues. See figure 6 below for the delays since the April 2013 replan to subcontractor Northrop Grumman’s delivery of the various cryocooler components to JPL or Goddard Space Flight Center.

As a result of NASA concerns with the sliding compressor assembly delivery date, the schedule for that component was again replanned in July 2014. However, the planned delivery date for the compressor assembly has already begun slipping. According to the cryocooler subcontractor program manager, the delivery schedule remains overly optimistic and meeting it would have required perfect execution of the remaining work. For example, according to project, JPL, and subcontractor officials, the July replan’s February 2015 subcontractor delivery date for the cryocooler compressor assembly was achievable only under a best case scenario where the subcontractor’s manufacturing issues did not persist. However, since July, issues such as a manufacturing error and manufacturing process oversight have delayed delivery until at least April 2015. For example, one delay occurred when a subcontractor employee conducted unauthorized work on one of the cryocooler compressors that led to a concern of possible compressor contamination. Ultimately, the subcontractor determined that no contamination occurred and the compressor was still usable—but not before a 3 week halt in work on the specific compressor for the investigation into the incident. JPL officials noted that the schedule impact would only be 8 days as they would be rearranging hardware flow and adding some holiday work to recover most of the time lost. Subsequently, a separate manufacturing error on a compressor assembly component resulted in 3 weeks of additional delay. In addition to these manufacturing issues, according to the subcontractor program manager, the subcontractor continues to encounter problems resulting from design immaturity of certain parts of the compressor assembly as they pursue the manufacturing effort.

Given the cryocooler subcontractor’s history of prior schedule delays and continued performance challenges, the project recognizes the cryocooler’s late delivery to JWST’s integration and testing schedules as a top issue to the project and is planning mitigation strategies in anticipation of further schedule slips—which would reduce the project’s overall schedule reserve further. JPL’s analysis of subcontractor performance trends on the compressor assembly schedule projects that the subcontractor is very unlikely to meet the current April 2015 delivery date and the compressor assembly is projected to be as late as November 2015, if past performance trends continue. The cryocooler used more schedule reserve than any JWST element or major subsystem in the past year, expending 5 months. Currently, the cryocooler schedule contains approximately 11 months of schedule reserve, of which 6 follow Consequently, without modifications spacecraft integration and testing.to the current integration and testing schedule, a subcontractor delivery of the compressor assembly to JPL after August 2015 would begin depleting the schedule reserve that follows spacecraft integration and testing. Through the reordering and compression of JPL’s test schedule following delivery of the compressor assembly, the project and JPL have accommodated cryocooler schedule slips to date. In anticipation of further delays to the cryocooler compressor assembly, the project and Northrop Grumman are exploring impact mitigation strategies in the event of JPL’s late delivery of the compressor assembly to spacecraft integration and testing, which is to begin in February 2016.

The cryocooler subcontractor has, however, resolved other significant challenges related to cryocooler design and valve leakage that, as we have previously reported, have been the basis of prior delays. For example, in 2014, a verification model for one of the cryocooler’s two compressor types was successfully tested, which confirmed that the design would achieve the temperatures required for the initial cooling stage—a key technical issue we reported on in 2014. Additionally, the project selected a valve for the cryocooler’s cold head assembly, the component of the cryocooler that allows it to cool MIRI to the 6 degree Kelvin temperature required for the instrument’s function.

Sunshield

Northrop Grumman experienced a significant manufacturing problem during the construction of the large composite panel that forms part of the sunshield’s primary support structure. This problem resulted in a loss of overall project schedule reserve and put the sunshield on the project’s critical path temporarily before it was overtaken by the cryocooler. Specifically, after the initial panel was manufactured, Northrop Grumman determined that it did not meet strength requirements. Because of the weakness identified in the panel, Northrop Grumman and NASA took some time to determine a root cause—moisture was absorbed into the panel during the manufacturing process—and implemented corrective actions for remanufacture of the panel. This total effort resulted in a loss of 1.75 months of total project schedule reserve. JWST officials expressed confidence in Northrop Grumman’s proposed remanufacture plan, which involves the incorporation of a moisture barrier to the tool upon which the panel is made to prevent moisture transfer. According to contractor officials, the replacement panel, which was manufactured using the modified process, was completed in October 2014 and a second, smaller panel is to be completed by January 2015. As early as 2010, the project and Northrop Grumman understood that the manufacturing process for the sunshield’s two composite panels would be complex and took risk mitigation steps, such as expedited manufacturing preparations, to address potential schedule impacts of this risk. Project officials maintain that these mitigation steps prevented the panel remanufacture from using the full 8 months of sunshield schedule reserve.

The contractor has made progress, however, on the manufacture of other components of the sunshield, such as the sunshield subsystem’s membrane layers and other structural hardware. Northrop Grumman successfully demonstrated the sunshield’s membrane deployment concept by conducting a full deployment test of a full scale engineering model of the membrane system in July 2014. Currently, one of the five flight membrane layers has been manufactured and work has begun on an additional two. Further, Northrop Grumman has completed manufacturing of the main beams of the sunshield’s primary support structure and tubes that assist in deploying the mid-boom structure.

ISIM and OTE Preparing for OTIS

The ISIM schedule has lost schedule reserve since our January 2014 report, and is a half month away from becoming the critical path of the project. The ISIM has used 3 months of schedule reserve out of the 7.5 months it held in 2013, due in part to longer than expected integration and testing for NIRCam and NIRSpec and to additional anticipated testing time for the second and third cryovacuum tests. The first of three ISIM cryovacuum tests—in which a test chamber is used to simulate the near absolute zero temperatures in space—was completed in November 2013, with two of the observatory’s four science instruments integrated. This first test was considered a risk reduction effort that provided insight into how to efficiently conduct the remaining two cryovacuum tests and how to analyze test results. The disruption of this first test from the government shutdown did result in some deferred testing, which increased the planned duration for the subsequent cryovacuum test. The second cryovacuum test, which included all four science instruments, began in June 2014 and finished in October 2014. According to project officials, the second ISIM cryovacuum test progressed as expected, with no technical disruptions. However, the schedule to prepare ISIM for the approximately 3.5-month-long third cryovacuum test—scheduled to commence in late July 2015—is considered very ambitious by NASA officials. Between the second and third cryovacuum tests, additional testing must be carried out, as well as the retrofitting of certain components, such as the near infrared detectors on two of the instruments.

The OTE has also used schedule reserve during preparations for the element’s integration with the ISIM. The OTE now has 4.5 months of schedule reserve remaining of the 7 months it held in 2013, putting it in the same position as ISIM in terms of currently held schedule reserve. Contractors successfully completed structural integrity testing of the OTE’s primary mirror backplane support structure that will hold the observatory’s science instruments and its 18-segment primary mirror. Additionally, for the 2015 readiness assessment of the chamber and test equipment for the subsequent integrated OTE and ISIM—known as OTIS—cryovacuum testing, the project is preparing a testing structure that incorporates a two-segment primary mirror and a secondary mirror. For this effort, the project took delivery of an engineering model of the OTE backplane’s center section at Goddard Space Flight Center in July 2014 and completed the process of mounting the two primary mirror segments.

Spacecraft

The spacecraft element also used 2.5 months of its schedule reserve over the past year, putting it a half month from the critical path. Specifically, the project reported that manufacturing challenges with the spacecraft structure required the use of schedule reserve to accommodate the delays. Despite these manufacturing challenges, the spacecraft completed significant milestones this year. Specifically, in January 2014, NASA assessed the spacecraft element as having successfully completed its critical design review, a major milestone indicating that the spacecraft’s design maturity was deemed appropriate to support proceeding with full-scale fabrication, assembly, integration, and test. While the review required actions to be taken, such as the effort to further mature certain technologies, most of these have been addressed, though work is ongoing on a few key spacecraft components. Among the technologies requiring further maturation is a release mechanism we reported on in January 2014, which was causing excessive shock when engaged. This mechanism has been modified and shock is now within an acceptable range, though testing is ongoing on the modified design.

The project continues to address other technical risks and challenges as it continues development and testing of the various JWST elements and major subsystems. For more information on selected risks and challenges, see appendix IV.

Project Lacks Current Cost Risk Analyses, Creating Uncertainty About Adequacy of Its Budget

The JWST project and prime contractor cost risk analyses have not been updated since the project’s replan in 2011 and do not account for newer risks. GAO best practices call for programs to regularly update cost risk analyses to account for new risks. We planned to conduct an independent updated cost risk analysis, but the prime contractor did not allow us the access necessary to execute our planned methodology. The project did subsequently agree to conduct its own cost risk analysis of the prime contractor’s remaining work. If properly conducted, its analysis will provide the project with the reliable information necessary to gauge whether the contractor’s budget is on target or at risk of future cost overruns.

Project and Prime Contractor Have Not Updated Risk Analyses to Account for New Risks

Neither NASA’s joint cost and schedule confidence level (JCL) analysis nor Northrop Grumman’s cost risk analysis to support the prime contract has been updated since 2011. Additionally, both NASA and Northrop Grumman reported that they do not intend to update their risk analyses, despite the fact that these analyses are 3 years old and almost 4 years remain until launch. As we have reported previously, GAO’s best practice criteria call for risk analyses to be updated regularly over the life of a project in order to ensure an estimate’s accuracy over time and account for actual costs and new project risks. Without regular updates to the cost risk analysis, JWST decision makers and stakeholders for the project cannot be certain that its existing cost estimates accurately portray the project’s financial status. Updating these analyses is particularly important in light of prior project optimism and newly discovered risks that threaten the project’s cost and schedule reserves.

Although our previous work found that the JWST project’s 2011 JCL was not fully reliable, the JWST project continues to rely on its original cost estimate, which was confirmed by the JCL, to provide the basis of its budget estimates and requests. Based on this estimate, the JWST project is currently reporting that the project remains on schedule to launch in October 2018 and within the $8 billion developmental cost cap designated by Congress. The JWST project manager stated that the project conducts a number of programmatic and analytical activities to monitor the health of its cost and schedule that he believes provides him with accurate information on the project’s status. In addition, the project conducted two schedule risk analyses in response to a recommendation we made in 2014, but analyses have not been completed for all schedules. The project stated one of its many methods of measuring cost and schedule health is analysis of earned value management (EVM) data. However, EVM analysis provides a cost and schedule projection based on past performance that does not reflect the potential impact of future risks. The 2011 JCL was the last risk analysis performed to model impacts of future risks on the project’s cost and schedule. In 2012 and 2014 we made both a recommendation to NASA and provided a matter to Congress for consideration related to the need to update the JWST JCL to address weaknesses we found and to incorporate new risk information. However, a new JCL has not been conducted to date and NASA has indicated it does not intend to update the JCL for JWST.

As a result of the significant changes that have occurred over the last 3 years since the 2011 replan and lacking reliable cost estimates, we determined that an updated cost risk analysis was warranted to provide Congress with insight on JWST’s remaining work. The changes we observed include risks that have been encountered both by JWST and Northrop Grumman that were not included in their 2011 cost risk analyses. For example, the highest risk for the JWST project in September 2014 was the delivery date of the cryocooler to mission integration and testing events, which was entered into the project’s risk database in January 2013. By October 2014, this risk had become one of the JWST project’s top issues and is threatening cost and schedule reserves. Additionally, 67 percent of the risks present in April 2014 on Northrop Grumman’s list of project risks were not present in September 2011. The JWST project’s budget system provides a means of quantifying future threats to the project’s budget by estimating a threat’s potential cost and likelihood of occurrence. However, our analysis of the system’s data since the replan found that almost 70 percent of reserve spending has not been anticipated as threats by the budget system. Without an analytical framework that comprehensively captures future risks into potential project costs, the project’s estimates do not represent a comprehensive portrayal of the financial status of the project. As a consequence, the project and Congress do not have all the data necessary to be confident in the project’s projected budgetary needs.

To address the lack of updated information, GAO planned to conduct an updated cost risk analysis on the Northrop Grumman prime contract, given that it represents the most significant contractor work remaining on the project. The methodology for this analysis, which has been cited as an important tool by NASA’s cost estimating community, would have provided an updated level of confidence associated with the project meeting its cost estimate and allowed decision makers to determine if additional cost reserves were warranted to mitigate future risks. However, we were unable to execute our methodology because Northrop Grumman declined to allow GAO to conduct an independent, unbiased analysis. GAO best practices for cost estimating require anonymous interviews with technical experts on a project to discuss the project’s risks. We were not able to create this environment because while Northrop Grumman officials were open to allowing us to conduct the analysis, they did not allow us to interview employees without a manager present. Anonymity is critical to the methodology in order to ensure the interviews are unbiased and provide a comprehensive portrayal of project risks.

The JWST project and NASA disagreed with the approach of an independent cost risk analysis conducted by an outside organization at this point in the project. Additionally, neither believes that an organization external to NASA can fully comprehend the project’s risks, and any such analysis would be overly conservative due to the complexities of the risks. As a result, they asserted that the results of any outside analysis would not be representative of the real risk posture of the project. GAO’s best practices call for cost estimates to be compared to independent cost estimates in addition to being regularly updated. Without an independent and updated analysis, both the committee members’ and NASA’s oversight and management of JWST will be constrained since the impact of newer risks have not been reflected in key tools, including the cost estimate.

In response to our concerns, NASA indicated it would conduct its own cost risk analysis for the committee members of the Northrop Grumman contract. The project already had plans to use additional EVM analysis, which the project does plan to use as a management tool, that incorporates risks to provide a health check on the project’s cost reserves. However, a NASA project official stated that data from the cost risk analysis was for the purpose of providing cost information to committee members, but the JWST project does not intend to use it to manage the project. Additionally, the project has indicated that it will consider updating the cost risk analysis in the future, but presently has not committed to regularly updating the analysis. To maintain quality cost estimates over the life of a project, best practices state that cost risk analyses be updated regularly to incorporate new risks and be used in conjunction with EVM analysis to validate cost estimates. If the project does not follow best practices in conducting its cost risk analysis or use it to manage the project, the resulting information may be unreliable and not substantively provide insight into JWST’s potential costs to allow either Congress or project officials to take any warranted action.

Contractors Have Been Responsive to Incentives, but Final Evaluation May Not Reflect Total Contract Performance

The JWST project has used NASA’s award fee structure to implement incentives that align with conferee’s priorities for cost and schedule for JWST and contractors have been responsive to these incentives, for example by making changes to key practices to improve communication. When completed, JWST’s prime contract will have spanned almost two decades, during which there have been and could be future significant variances in contractor performance and key management changes among program officials. While NASA’s award fee guidance and its procurement regulations require the development of explicit evaluation factors to evaluate performance, evaluation criteria are not specified for the final evaluation of total contract performance in the project’s performance evaluation plans. Without clear criteria in award fee plans, it is unclear how JWST officials will assess contractor performance over the life of the contracts, as is required per NASA regulations—particularly for the prime contract, which will include a significant cost overrun, three launch delays, and almost 20 years of varying performance.

NASA’s Cost-Plus-Award- Fee Process to Assess Contractor Performance

Award fee contracts provide contractors the opportunity to obtain monetary incentives for performance in designated areas, which can be weighted differently at NASA’s discretion. For JWST’s award fee contracts, these areas include cost, schedule, technical, and business management and are established in the contracts’ performance evaluation plans. Award fees are used when key elements of performance cannot be defined objectively, and, as such, require the project officials’ judgment to assess contractor performance. Table 1 provides a general outline of the performance evaluation process.

Contractors Responsive to NASA’s Use of Cost and Schedule Incentives

The JWST project has used NASA’s award fee structure to implement incentives that align with conferee’s priorities for cost and schedule for JWST. The JWST project has modified the fee structure for one of its two cost-plus-award-fee contracts, the Northrop Grumman prime contract, to enhance cost and schedule incentives. In December 2013, the JWST project completed contract negotiations with Northrop Grumman to incorporate the launch delay to October 2018. During negotiations, NASA and Northrop Grumman agreed to replace a $56 million on-orbit incentive fee with award fees to increase the available award fee for the entire contract to almost a quarter of a billion dollars. While award fees for the Northrop Grumman contract are intended to reward performance for cost, schedule, and meeting technical requirements, the on-orbit incentive would have been based on exceeding technical requirements. Both JWST and Northrop Grumman officials stated restructuring the award fee incentives gave NASA more flexibility to incentivize the contractor to prioritize cost and schedule performance over exceeding technical requirements. In addition, a $6 million schedule incentive fee was added to the contract that would be paid to the contractor for achieving three key deliveries ahead of schedule.modifications were intended to help ensure the Northrop Grumman contract contained proper cost and schedule incentives.

According to NASA officials, these For JWST’s two award fee contracts—Northrop Grumman and Exelis— NASA award fee letters of award fee periods from February 2013 to March 2014 indicate that contractors have been responsive to interim award fee period criteria provided by NASA. Also, contractor officials from both award fee contracts confirmed that they pay close attention to this guidance in prioritizing their work. For example, Northrop Grumman officials reported that they had made specific changes to improve communications in direct response to this guidance which was validated by award fee letters from NASA.

History of Varied Performance Is Not Included in Criteria for Final Evaluation of Contractor Performance

The JWST project does not have sufficient award fee guidance for two of its largest contracts, as required by the NASA Federal Acquisition Regulation Supplement. Our finding is consistent with the NASA Office of Inspector General findings that cost-plus-award-fee contracts did not include explicit criteria related to the final evaluation.assessing Northrop Grumman’s prime contract do not specify that in the final evaluation performance metrics for cost and schedule should reflect performance prior to the replan. If performance prior to the replan is not considered, the contractor could receive award fee that does not reflect its performance over the life of the contract. To see how JWST’s cost- plus-award-fee contracts compare to NASA policy and guidance for both contracts, see table 2.

Conclusions

As the JWST project enters fiscal year 2015 with under 4 years until launch, project officials have made progress building significant pieces of hardware, but continue to face numerous technical challenges that have eroded schedule reserve for the first time since the 2011 replan. While some use of schedule reserve is expected, JWST is a schedule-driven program and preserving the reserve is critical. The project will have fewer opportunities for the types of workarounds it has been able to incorporate thus far because initiating work is often dependent on the successful and timely completion of the prior work. Further, the project has limited funds available in the short-term to use to preserve schedule. Effectively managing and making targeted use of existing cost reserves to preserve schedule reserve until fiscal year 2016, when more reserves are available, is critical to maintaining the planned launch date in October 2018.

Three years after the 2011 replan, project officials and the prime contractor have not recently assessed risks for the potential impact to cost to ensure they are measuring progress against a valid baseline. While the project reports that JWST remains on schedule and within its developmental cost cap, this is based on outdated analysis and risks that were present in 2011. Without incorporating the myriad of new and sometimes significant risks into any assessment of cost risk, the project’s cost risk posture cannot be independently validated. While NASA has chosen to conduct a cost risk analysis of Northrop Grumman’s remaining work itself, its quality hinges on whether NASA chooses to follow best practices. Doing so would better ensure that the results of the analysis would be reliable and would provide an accurate portrayal of status that can then be used to ensure better informed decisions are being made about project funding forward, at least for Northrop Grumman’s remaining work. Furthermore, because JWST is one of NASA’s most expensive and complex science projects and any issues can have significant impacts to other science projects, going above and beyond what is required per policy or guidance is one small step that could help demonstrate NASA’s commitment to the success of JWST and to improving acquisition management outcomes.

Effectively using the award fee evaluation process to thoroughly and fairly assess contractor performance is critical given the size and long duration of the JWST contracts, in particular its prime contract with Northrop Grumman. Without clear guidance for both NASA officials and contractors on how to conduct the final award fee evaluation and account for performance over the life of the project, the final determination of all award fees for two of JWST’s largest contracts may not accurately reflect the contractor’s performance. Fairly assessing that performance over the life of the contract is especially critical for JWST given the variance in performance that Northrop Grumman has shown from contract award in 2002 to the present. Not doing so could result in Northrop Grumman receiving a higher percentage of award fee than interim scores would warrant.

Recommendations for Executive Action

We recommend that the NASA Administrator take the following two actions: In order to provide additional information and analyses to effectively manage the program and account for new risks identified after the 2011 replan, direct JWST project officials to follow best practices while conducting a cost risk analysis on the prime contract for the work remaining and ensure the analysis is updated as significant risks emerge.

In order to ensure JWST’s award fee contracts’ final evaluations thoroughly and fairly evaluate contractor performance over the life of the contract and to provide clarity to the process that will be used for the final evaluation, direct JWST project officials, in conjunction with the performance evaluation board for JWST and the Goddard Space Flight Center fee determining official, to modify performance evaluation plans for its award fee contracts to ensure they (a) specify evaluation criteria that reflects total contract performance in advance of the final evaluation, and (b) clearly describe the process the performance evaluation board and fee determining official will use to evaluate contractor performance in the final evaluation.

Agency Comments and Our Evaluation

NASA provided written comments on a draft of this report. These comments are reprinted in appendix V. In responding to a draft of this report, NASA partially concurred with one recommendation and concurred with another.

NASA partially concurred with our recommendation directing the JWST project to follow best practices while conducting a cost risk analysis on the prime contract for the work remaining and ensure it is updated as significant risks emerge. In response to this recommendation, NASA stated that the JWST program and project use a range of tools to assess all major contractors’ performance and that the approach the project has in place is consistent with best practices. NASA indicated that it will continue to use the tools currently in place to manage JWST. Furthermore, NASA noted the JWST project initiated a cost risk analysis of the prime contract that is incorporating best practices and will update it when required by NASA policy.

NASA policy did not require cost risk analyses when the JWST project began, therefore the project is also not required to update such an analysis. Policy notwithstanding, NASA and GAO’s best practices do support conducting a cost risk analysis and updating it as product development progresses. Specifically, NASA's cost estimating handbook states that while the product is being designed, developed, and tested, there are changes which can impact the estimate and the risk assessment. It is critical to capture these changes to maintain a realistic program estimate now and in the future. During this phase, programmatic data may have just as much of an impact on the estimate and risk assessment as technical data. We acknowledge in the report the JWST project’s use of EVM data, but reiterate that it does not reflect the potential impact of future risks. As our report notes, the JWST project has only started two of the five integration and test periods and risks have changed significantly since a cost risk analysis was last conducted on the prime contractor's work. As the telescope is one of NASA's most complex and expensive projects, both JWST and NASA officials have a tremendous stake in the success of the JWST project. As a result, an important best practice NASA can follow is to conduct a cost risk analysis in the future so that realistic estimates are being communicated to Congress, especially as new risks emerge in future testing. We encourage JWST officials to rely not on policy direction but on good management practices to spur them to update the cost risk analysis they are conducting on the prime contractor's work as significant risks continue to emerge.

NASA concurred with our recommendation directing the JWST project, in conjunction with the performance evaluation board for JWST and the Goddard Space Flight fee determining official, to modify its performance evaluation plans for its award fees to ensure they (a) specify evaluation criteria that reflects total contract performance in advance of the final evaluation, and (b) clearly describe the process the performance evaluation board and fee determining official will use to evaluate contractor performance in the final evaluation. NASA indicated it will revise the performance evaluations plans to clarify that criteria in the existing plans applies to both interim periods and the final award fee determination and will revise the plans to specify the process that will be used to conduct the final evaluation and final determination. Once complete, these actions should address our recommendation to clarify the criteria and the process that will be used so a thorough and fair final evaluation may occur that incorporates information from performance over the life of each contract.

NASA also provided technical comments, which have been addressed in the report, as appropriate.

We are sending copies of the report to NASA’s Administrator and interested congressional committees. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

Should you or your staff have any questions on matters discussed in this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to assess the extent to which (1) technical challenges are impacting the James Webb Space Telescope (JWST) project’s ability to stay on schedule and budget, (2) budget and cost estimates reflect current information about project risks, and (3) the project uses award fee contracts to motivate and assess contractor performance.

To assess the extent to which technical challenges are impacting the JWST project’s ability to stay on schedule and budget, we reviewed project and contractor schedule and cost documentation, analyzed the progress made against planned milestones established during the project’s replan in 2011, and held interviews with program, project, and contractor officials. We reviewed project monthly status reviews, quarterly provided flight program reviews, information from the project’s risk database, and other technical documentation to gain insights on the project’s progress since our last report in January 2014. In addition, we reviewed briefings and schedule documentation provided by project and contractor officials. These documents included information on the project’s technological challenges and risks, mitigation plans, timelines for addressing these risks and challenges, and available schedule reserve. We analyzed the project’s reported schedule reserve data across multiple elements and major subsystems. We also interviewed project officials to clarify information and to obtain additional information on element and major subsystem level risks and technological challenges. Further, we attended flight program reviews at the National Aeronautics and Space Administration (NASA) headquarters on a quarterly basis where the current status of the program was briefed to NASA headquarters officials outside of the project. Finally, we interviewed officials from Exelis, the Jet Propulsion Laboratory, and different divisions of Northrop Grumman Aerospace Systems concerning risks and challenges on the elements and major subsystems, instruments, or components they were developing.

To assess the extent to which budget and cost estimates reflect current information about project risks, we reviewed project, contractor, and subcontractor documentation and held interviews with officials from all of these different organizations. We reviewed project monthly status reviews, quarterly provided flight program reviews, and contractor information on the potential cost to address identified risks. To identify whether JWST’s current cost management practices were sufficient, we reviewed its earned value management (EVM) and budget systems and compared them to GAO best practices.To identify new risks that had emerged since the 2011 replan, we conducted analysis to compare the prime contractor’s current risk documentation to the risk documentation at the time of the 2011 replan. Further, we analyzed the project’s cost reserve data since the 2011 replan to identify where the project has expended cost reserves and predicted threats, and conducted interviews with contractors to discuss their cost reserve management practices. We also examined and analyzed EVM data from several contractors to identify trends in cost and schedule performance, entering both high-level and subsystem-level monthly contractor EVM data into a GAO-developed spreadsheet, which includes checks to ensure the EVM data provided was reliable. We discussed our assessments of the project’s data and analysis with JWST program and project officials from the Goddard Space Flight Center.

To assess the extent to which the project uses award fee contracts to motivate and assess contractor performance, we reviewed the project’s documentation, conducted contract file reviews, examined performance evaluation plans for two contractors, and reviewed the NASA Office of Inspector General’s report on NASA’s use of award fees. examined the relevant portions related to award fees of the Federal Acquisition Regulation (FAR), NASA FAR Supplement, and NASA Award Fee Contracting Guide. We conducted interviews with project officials, NASA Office of Procurement officials, the JWST fee determining official, and contractor officials to obtain information on the project’s use of contract incentives and implementation of award fee evaluations. We evaluated the adequacy of JWST’s performance evaluation criteria and processes by comparing the processes and criteria in the project’s performance evaluation plans and other documentation to NASA’s policies, regulations, and guidance. We interviewed project and NASA headquarters officials to clarify information and obtain additional information on NASA’s typical award fee processes and plans for evaluating JWST’s contractors.

NASA, Office of Inspector General, NASA’s Use of Award Fee Contracts, IG-14-003 (Washington, D.C.: Nov. 19, 2013).

Our work was performed primarily at NASA headquarters in Washington, D.C., Goddard Space Flight Center in Greenbelt, Maryland, and Johnson Space Center in Houston, Texas. We also visited the Jet Propulsion Laboratory in Pasadena, California; Northrop Grumman Aerospace Systems in Redondo Beach, California; the Defense Contract Management Agency in Redondo Beach, California; and the Space Telescope Science Institute in Baltimore, Maryland.

We conducted this performance audit from February 2014 to December 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Elements and Major Subsystems of the James Webb Space Telescope (JWST) Observatory

Appendix III: Organizational Chart for the James Webb Space Telescope Program

Appendix III: Organizational Chart for the James Webb Space Telescope Program The Jet Propulsion Laboratory is the contractor for the development of the cryocooler, but has subcontracted most of the work to a different division of Northrop Grumman than the one that is responsible for OTE, spacecraft, and sunshield development.

Appendix IV: Other Technical Issues for the James Webb Space Telescope

Appendix V: Comments from the National Aeronautics and Space Administration

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Cristina Chaplain, (202) 512-4841 or chaplainc@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Shelby S. Oakley, Assistant Director; Karen Richey, Assistant Director; Jason Lee, Assistant Director; Patrick Breiding; Laura Greifner; Samuel P. Harris; Jonathan Mulcare; Kenneth E. Patton; Timothy N. Shaw; Sylvia Schatz; Jay Tallon; and Ozzy Trevino made key contributions to this report.