JAMES WEBB 

SPACE TELESCOPE

Project on Track but 

May Benefit from 

Improved Contractor 

Data to Better 

Understand Costs

Report to Congressional Committees

December 2015

GAO-16-112

United States Government Accountability Office
United States Government Accountability Office

Highlights of GAO-16-112, a report to 

congressional committees

December 2015

JAMES WEBB SPACE TELESCOPE

Project on Track but May Benefit from Improved 

Contractor Data to Better Understand Costs

Why GAO Did This Study

JWST is one of NASA’s most complex 

and expensive projects, at an 

anticipated cost of $8.8 billion. With 

significant integration and testing 

scheduled in the 3 remaining years 

until the planned launch date, the 

JWST project will need to continue to 

address many challenges and identify 

problems, many likely to be revealed 

during its rigorous testing to come. The 

continued success of JWST hinges on 

NASA’s ability to anticipate, identify, 

and respond to these challenges in a 

timely and cost-effective manner to 

meet its commitments.

Conference Report 112-284 included a 

provision for GAO to assess the project 

annually and report on its progress. 

This is the fourth such report. This 

report assesses (1) the extent to which 

JWST is meeting its schedule 

commitments and (2) the current cost 

status of the project, among other 

issues. To conduct this work, GAO 

reviewed monthly JWST reports, 

reviewed relevant policies, conducted 

independent analysis of NASA and 

contractor data, and interviewed NASA 

and contractor officials.

What GAO Recommends

GAO recommends that the JWST project 

require contractors to identify, explain, 

and document anomalies in contractordelivered monthly earned value 

management reports. GAO continues to 

believe that its 2012 recommendation to 

implement formal surveillance to help 

improve the reliability of contractorprovided data has merit and should be 

implemented. NASA concurred with the 

recommendation made in this report. 

What GAO Found

The National Aeronautics and Space Administration’s (NASA) James Webb 

Space Telescope (JWST) project is meeting its schedule commitments, but it will 

soon face some of its most challenging integration and testing. JWST currently 

has almost 9 months of schedule reserve—down more than 2 months since

GAO’s last report in December 2014—but still above its schedule plan and the 

Goddard Space Flight Center requirement. However, as GAO also found in 

December 2014, all JWST elements and major subsystems continue to remain 

within weeks of becoming the critical path—the schedule with the least amount of 

schedule reserve—for the overall project. Given their proximity to the critical 

path, the use of additional reserve on any element or major subsystem may 

reduce the overall project schedule reserve. 

James Webb Space Telescope Schedule Reserve Remaining in 2014 and 2015

Before the planned launch in October 2018, the project must complete five major 

integration and test events, three of which have not yet begun. Integration and 

testing is when problems are often identified and schedules tend to slip. At the 

same time, the project must also address over 100 technical risks and ensure 

that potential areas for mission failure are fully tested and understood.

JWST continues to meet its cost commitments, but unreliable contractor 

performance data may pose a risk to project management. To help manage the 

project and account for new risks, project officials conducted a cost risk analysis 

of the prime contract. A cost risk analysis uses information about cost drivers, 

technical issues, and schedule to determine the reliability of a program’s cost 

estimates. GAO found that while NASA’s cost risk analysis substantially met best 

practices for cost estimating, officials do not plan to periodically update it. 

Instead, the project is using a risk-adjusted analysis to update and inform its cost 

position, but this analysis is a simplified version of a cost risk analysis—and not a 

replacement—and is based on contractor-provided performance data that 

contains anomalies that render the data unreliable. Further, the project does not 

have an independent surveillance mechanism, such as the Defense Contract 

Management Agency, to help ensure data anomalies are corrected by the 

contractor before being incorporated into larger cost analyses, as GAO

recommended in 2012. As a result, the project is relying partially on unreliable 

information to inform its decision making and overall cost status.

View GAO-16-112. For more information, 

contact Cristina Chaplain at (202) 512-4841 or 

chaplainc@gao.gov.
Page i GAO-16-112 James Webb Space Telescope

Letter 1

Background 3

JWST Is Meeting Schedule Commitment with Majority of Testing 

and Challenging Integration Work Still to Come 11

Project Continues to Meet Cost Commitments but Contractor 

Workforce Increases and Unreliable Contractor Performance 

Data Pose Management Challenges 20

Independent Oversight Has Been Built into JWST’s Development 

Life Cycle to Improve the Likelihood of Mission Success 29

Conclusions 35

Recommendation for Executive Action 36

Agency Comments and Our Evaluation 36

Appendix I: Objectives, Scope, and Methodology 38

Appendix II: Elements and Major Subsystems of the James Webb Space 

Telescope (JWST) Observatory 42

Appendix III: Organizational Chart for the James Webb Space Telescope (JWST) Program 43

Appendix IV: Comments from the National Aeronautics and Space Administration 44

Appendix V: GAO Contact and Staff Acknowledgements 46

Tables

Table 1: Tests Removed from Cryocooler Acceptance and End-toEnd Testing 16

Table 2: Standing Review Board Types at the National 

Aeronautics and Space Administration 31

Contents
Page ii GAO-16-112 James Webb Space Telescope

Figures

Figure 1: James Webb Space Telescope 5

Figure 2: Integration and Test (I&T) Schedule for the James Webb 

Space Telescope Elements 6

Figure 3: Components of the Mid-Infrared Instrument Cryocooler 8

Figure 4: Schedule Reserve Held by Each Element and Major 

Subsystem for the James Webb Space Telescope in 

2014 and 2015 12

Figure 5: Cryocooler Subsystem Integration and Test 17

Figure 6: Northrop Grumman Workforce Growth in Fiscal Years 

2014 and 2015, Projections versus Actuals at Beginning 

of Fiscal Year 23
Page iii GAO-16-112 James Webb Space Telescope

Abbreviations 

EAC estimate at completion 

EVM earned value management

FGS Fine Guidance Sensor

Goddard Goddard Space Flight Center

I&T integration and test

ICRP Independent Comprehensive Review Panel

ISIM Integrated Science Instrument Module

JPL Jet Propulsion Laboratory

JWST James Webb Space Telescope

K Kelvin

MIRI Mid-Infrared Instrument

NASA National Aeronautics and Space Administration

NIRCam Near-Infrared Camera

NIRISS Near-Infrared Imager and Slitless Spectrograph

NIRSpec Near-Infrared Spectrograph

OTE Optical Telescope Element

OTIS Optical Telescope Element and Integrated 

Science Instrument Module

RFA Request For Action 

STScI Space Telescope Science Institute

IV&V Independent Verification and Validation

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-16-112 James Webb Space Telescope

441 G St. N.W.

Washington, DC 20548

December 17, 2015

Congressional Committees

The James Webb Space Telescope (JWST) is one of the National 

Aeronautics and Space Administration’s (NASA) most complex projects 

and top priorities. The innovative technologies within the telescope as 

well as the sheer size of some of its components—such as the tenniscourt-sized sunshield—illustrate some of the immense challenges in 

building it. While JWST has been an idea discussed and planned for 

more than two decades, major pieces of the telescope are being built, 

tested, and prepared for integration. The project has now entered into a 

significant integration and test period that is expected to last until the 

telescope is launched in the fall of 2018. It is in this period when elements 

and major subsystems are integrated and tested in which unforeseen 

challenges could arise and affect the cost and schedule for the project. 

Until launch, NASA and its contractors’ ability to identify and respond to 

future challenges in a timely and cost-effective manner will likely influence 

whether JWST can meet its cost and schedule commitments to 

Congress.

The on-time and on-budget delivery of JWST is also a high congressional 

priority, as Conference Report 112-284 included a provision for GAO to 

assess the JWST program annually and to report to the Committees on 

Appropriations on key issues relating to program and risk management, 

achievement of cost and schedule goals, program technical status, and 

oversight mechanisms.1 This report is our fourth in response to that 

provision. For this report, we assessed: (1) the extent to which technical 

challenges have impacted the JWST project’s ability to meet its schedule 

commitments; (2) the current cost status of the JWST project and the 

primary challenges that may influence the project’s ability to meet future 

cost commitments; and (3) the extent to which independent oversight 

provides insight about project risks to management.

Our approach included an examination of the schedule, technical, and 

cost performance of the project since our last report in December 2014—

 1

H.R. Rep. No. 112-284, at 254 (2011) (Conf. Rep.).

Letter
Page 2 GAO-16-112 James Webb Space Telescope

which focused on the project’s cost and schedule commitments, the 

project’s lack of updated cost risk analyses for its major contractors, and 

the project’s use of award fees to manage contractors—as well as the 

extent to which independent oversight mechanisms are in place.2 To 

assess the extent to which technical challenges have impacted the JWST 

project’s ability to meet its schedule commitments, we reviewed project 

and contractor schedule documentation, monthly status reports, selected 

individual risks from monthly risk registers, previous and current test 

schedules, and other documentation; and held interviews with program, 

project, and contractor officials on the progress made and challenges 

faced building the different components of the telescope. We also 

interviewed experts within and outside of NASA to identify criteria, best 

practices, and metrics that could be used to assess the project’s progress 

in reducing risk or provide insight into the health of the project.

To assess the current cost status of the JWST project and the primary 

challenges that influence the project’s ability to meet its future cost 

commitments, we analyzed program, project, contractor, and 

subcontractor cost data and documentation. We compared projected 

workforce levels to actual workforce levels to determine differences and 

their effect on cost. We also conducted an analysis on the earned value 

management (EVM) data to ensure the reliability of the data over a 17-

month period for two of the project’s contractors as discussed later in the 

report.3 In addition, we assessed NASA’s 2014 cost risk analysis of 

Northrop Grumman’s remaining work to determine whether it followed 

best practices—such as whether modeling probability distributions were 

based on data availability, reliability, and variability; simulations were run 

to obtain a distribution of possible cost outcomes; and risk management 

plans were implemented, among others—and how these and other tools 

 2

GAO, James Webb Space Telescope: Project Facing Increased Schedule Risk with 

Significant Work Remaining, GAO-15-100 (Washington, D.C.: Dec. 15, 2014).

3

EVM is a project management tool that integrates the technical scope of work with 

schedule and cost elements for investment planning and control. It compares the value of 

work accomplished in a given period with the value of the work expected in that period. 

Differences in expectations are measured in both cost and schedule variances. The Office 

of Management and Budget requires agencies to use EVM in their performance-based 

management systems for the parts of an investment in which development effort is 

required or system improvements are under way. We selected the two contractors that 

use EVM and have the largest amount of work remaining.
Page 3 GAO-16-112 James Webb Space Telescope

informed the project’s cost status.4 Appendix I identifies all of the 

applicable best practices we used to assess the cost risk analysis.

To assess the extent to which independent oversight provided insight to 

management about project risks, we reviewed NASA policies and 

guidance documents to understand the elements for setting up and 

managing a Standing Review Board; interviewed Independent Program 

Assessment Office officials and past and current Standing Review Board 

members; collected information and analysis from NASA’s Independent 

Verification and Validation (IV&V) group; and interviewed IV&V officials. 

See appendix I for a detailed description of our objectives, scope, and 

methodology.

We conducted this performance audit from February 2015 to December 

2015 in accordance with generally accepted government auditing 

standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objectives. We believe 

that the evidence obtained provides a reasonable basis for our findings 

and conclusions based on our audit objectives.

JWST is envisioned to be a large deployable, infrared-optimized space 

telescope and the scientific successor to the aging Hubble Space 

Telescope. JWST is being designed for a 5-year mission to find the first 

stars and trace the evolution of galaxies from their beginning to their 

current formation, and is intended to operate in an orbit approximately 1.5 

million kilometers—or 1 million miles—from the Earth. With a 6.5-meter 

primary mirror, JWST is expected to operate at about 100 times the 

sensitivity of the Hubble Space Telescope. JWST’s science instruments 

are to observe very faint infrared sources and as such are required to 

operate at extremely cold temperatures. To help keep these instruments 

cold, a multi-layered tennis-court-sized sunshield is being developed to 

protect the mirrors and instruments from the sun’s heat. The sunshield 

and primary mirror are designed to fold and stow for launch and fit within 

the launch vehicle. When complete, the observatory segment of JWST is 

to include several elements (Optical Telescope Element (OTE), 

 4

GAO, GAO Cost Estimating and Assessment Guide, GAO-09-3SP, (Washington, D.C.: 

Mar. 2, 2009).

Background
Page 4 GAO-16-112 James Webb Space Telescope

Integrated Science Instrument Module (ISIM), and spacecraft) and major 

subsystems (sunshield and cryocooler). The JWST project is divided into 

three major segments: the observatory segment, the ground segment, 

and the launch segment. The hardware configuration created when the 

Optical Telescope Element and the Integrated Science Instrument 

Module are integrated, referred to as OTIS, is not considered an element 

by NASA, but we categorize it as such for ease of discussion. 

Additionally, JWST is dependent on software to deploy and control 

various components of the telescope as well as collect and transmit data 

back to Earth. The elements, major subsystems, and software are being 

developed through a mixture of NASA, contractor, and international 

partner efforts. See figure 1 below for an interactive graphic that depicts 

the elements and major subsystems of JWST.5 For more information on 

JWST’s organizational structure, see appendix III.

 5

The ground segment is not pictured in figure 1, the interactive graphic.
Page GAO-16-112 James Webb Space Telescope

Figure 1: James Webb Space Telescope

Sources: GAO (analysis); National Aeronautics and Space Administration (NASA) (data and images). | GAO-16-112

Interactive Graphic Rollover the white dots to see description. See appendix II for the non-interactive, printer-friendly version.

Integrated Science

Instrument Module

Optical 

Telescope

Element

Spacecraft

Sunshield

Optical Telescope &

Integrated Science

Instrument Module
Page 6 GAO-16-112 James Webb Space Telescope

Given JWST’s complexity, integration and test activities are comprised of 

five separate periods—two of which have already started—over the 

course of almost 7 years to build the observatory. During the test periods, 

the project works to mitigate risks to an acceptable level prior to launch. 

According to project officials, while some risks may be eliminated entirely 

through various mitigation strategies, others will be accepted as residual 

risks that remain upon launch. See figure 2 below for the overall planned 

integration and test flow for JWST that includes the remaining schedule 

reserve—or extra time built into the schedule to address any issues 

found.

Figure 2: Integration and Test (I&T) Schedule for the James Webb Space Telescope Elements

Note: There are multiple lower level I&T efforts that flow in to the ISIM, OTE, and Spacecraft I&T flow 

that are not depicted on figure 2. For example, the sunshield is a major subsystem that has its own 

I&T effort that runs parallel to Spacecraft I&T and ends with delivery to Spacecraft I&T.

For the majority of the work remaining, the JWST project will rely on three 

contractors: Northrop Grumman, Harris (formerly Exelis), and the Space 

Telescope Science Institute (STScI). Northrop Grumman plays the largest 

role, developing the sunshield, the OTE, the spacecraft, and a cooling 

subsystem for the Mid-Infrared Instrument (MIRI). Northrop Grumman 
Page 7 GAO-16-112 James Webb Space Telescope

performs most of this work under a prime contract with NASA, but its work 

on the MIRI cooler is performed under a separate subcontract with the Jet 

Propulsion Laboratory (JPL). Harris is manufacturing the test equipment, 

equipping the test chamber, and assisting in the testing of the optics of 

JWST. Finally, STScI will collect and evaluate research proposals from 

the scientific community and will receive and store the scientific data 

collected, both of which are services that they currently provide for the 

Hubble Space Telescope. Additionally, STScI is responsible for 

developing the ground system that manages and controls the telescope’s 

observations on behalf of NASA.

The MIRI instrument, one of the four instruments within ISIM, requires a 

dedicated, interdependent two-stage cooler subsystem designed to cool 

the infrared light detector to about 6.7 Kelvin (K), just above absolute 

zero. This cooler is referred to as a cryocooler and works by moving 

helium gas through about 10 meters (approximately 33 feet) of refrigerant 

lines located on the sun-facing surface of the JWST observatory to the 

colder, shaded side where the ISIM is located. According to NASA 

officials, a cryocooler of this configuration has never been developed or 

flown in space before. See figure 3 below for an illustration of the MIRI 

cryocooler on JWST and the varying temperatures needed in different 

areas of the telescope.
Page 8 GAO-16-112 James Webb Space Telescope

Figure 3: Components of the Mid-Infrared Instrument Cryocooler

Complex development efforts like JWST face myriad risks and 

unforeseen technical challenges which oftentimes can become apparent 

during integration and testing. To accommodate these risks and 

unknowns, projects reserve extra time in their schedules—which is 

referred to as schedule reserve—and extra money in their budgets—

which is referred to as cost reserve. Schedule reserve is allocated to 

specific activities, elements, and major subsystems in the event there are 

delays or to address unforeseen risks. Each JWST element and major 

subsystem has been allocated schedule reserve. When an element or 

major subsystem exhausts schedule reserve, it may begin to affect 

schedule reserve on other elements or major subsystems whose 

progress is dependent on prior work being finished for its activities to 

proceed. The element or major subsystem with the least amount of 

schedule reserve determines the critical path for the project. Any delay to 

Cost and Schedule 

Reserves for NASA 

Projects
Page 9 GAO-16-112 James Webb Space Telescope

an activity that is on the critical path will reduce schedule reserve for the 

whole project, and could ultimately impact the overall project schedule.

Cost reserves are additional funds within the project manager’s budget 

that can be used to address unanticipated issues for any element or 

major subsystem and are used to mitigate issues during the development 

of a project. For example, cost reserves can be used to buy additional 

materials to replace a component or, if a project needs to preserve 

schedule reserve, reserves can be used to accelerate work by adding 

shifts to expedite manufacturing and save time. NASA’s Goddard Space 

Flight Center (Goddard)—the NASA center with responsibility for 

managing JWST—has issued procedural requirements that establish the 

levels of both cost and schedule reserves that projects must hold at 

project confirmation.6 After this point, a specified amount of schedule 

reserve continues to be required throughout the remainder of 

development.7 In addition to cost reserves held by the project manager, 

management reserves are funds held by the contractors that allow them 

to address cost increases throughout development. We have found that 

management reserves should contain 10 percent or more on the cost to 

complete a project and are used to address different issues.8

JWST has experienced significant increases to project costs and 

schedule delays. Prior to being approved for development, cost estimates 

of the project ranged from $1 billion to $3.5 billion with expected launch 

dates ranging from 2007 to 2011. Before 2011, early technical and 

management challenges, contractor performance issues, low level cost 

reserves, and poorly phased funding levels caused JWST to delay work 

after confirmation, which contributed to significant cost and schedule 

overruns, including launch delays. The Chair of the Senate Subcommittee 

on Commerce, Justice, Science, and Related Agencies requested from 

NASA an independent review of JWST in June 2010. In response, NASA 

 6

The formulation phase culminates in a Key Decision Point C review known as project

confirmation, where cost and schedule baselines are to be established and documented in 

the agency baseline commitment. Project progress can subsequently be measured 

against these baselines. 

7

Goddard Space Flight Center, Goddard Procedural Requirements 7120.7 (May 4, 2008).

8

GAO, NASA: Earned Value Management Implementation across Major Spaceflight 

Projects Is Uneven, GAO-13-22 (Washington, D.C.: Nov. 19, 2012); GAO-09-3SP. 

History of Cost Growth, 

Low Project Reserves, 

and Schedule Delays
Page 10 GAO-16-112 James Webb Space Telescope

commissioned the Independent Comprehensive Review Panel, which 

issued its report in October 2010, and concluded that JWST was 

executing well from a technical standpoint, but that the baseline funding 

did not reflect the most probable cost with adequate reserves in each 

year of project execution, resulting in an unexecutable project.9 Following 

this review, the JWST program underwent a replan in September 2011, 

and Congress in November 2011 placed an $8 billion cap on the 

formulation and development costs for the project.10 On the basis of the 

replan, NASA rebaselined JWST with a life-cycle cost estimate of $8.835 

billion that included additional money for operations and a planned launch 

in October 2018. The revised life-cycle cost estimate included a total of 

13 months of funded schedule reserve. In the President’s fiscal year 2013 

budget request, NASA reported a 66 percent joint cost and schedule 

confidence level—lower than the 70 percent level noted in NASA 

procedural requirements—for these cost and schedule baselines.11 A joint 

cost and schedule confidence level is the process NASA uses to assign a 

percentage to the probable success of meeting cost and schedule 

estimates and is part of the project’s estimating process.

 9

James Webb Space Telescope (JWST) Independent Comprehensive Review Panel 

(ICRP): Final Report (Oct. 29, 2010). 

10During formulation, the basic project concept is defined, as well as the technologies, 

time, funding, and other resource requirements. 

11The joint cost and schedule confidence level is a quantitative probability analysis that 

requires the project to combine its cost, schedule, and risks into a complete quantitative 

picture to help assess whether the project will be successfully completed within cost and 

on schedule. NASA introduced the analysis in 2009, and it is among the agency’s 

initiatives to reduce acquisition management risk. The move to probabilistic estimating 

marks a major departure from NASA’s prior practice of establishing a point estimate and 

adding a percentage on top of that point estimate to provide for contingencies. NASA’s 

procedural requirements state that Mission Directorates should plan and budget programs 

and projects based on a 70 percent joint cost and schedule confidence level, or at a 

different level as approved by the Decision Authority of the Agency Program Management 

Council, and any joint cost and schedule confidence level approved at less than 70 

percent must be justified and documented. NASA Procedural Requirements (NPR) 

7120.5E, NASA Space Flight Program and Project Management Requirements, 

paragraph 2.4.4 and 2.4.4.1 (Aug. 14, 2012). 
Page 11 GAO-16-112 James Webb Space Telescope

In December 2014, we found that the project was progressing within the 

2011 replan for both cost and schedule.12 We reported on technical 

challenges with JWST elements and major subsystems that had 

consumed a portion of the cost and schedule reserves. We also found 

that the cryocooler remained an ongoing challenge and continued to use 

a disproportionate amount of cost reserves. Finally, we found that NASA 

had not conducted a cost risk analysis since the 2011 replan. A cost risk 

analysis determines the reliability of a program’s cost estimate by 

determining a program’s cost drivers and the risk of cost overruns through 

an analysis that links historical schedule information along with technical 

issues and uncertainties in schedule and cost. Since new risks had 

emerged, we recommended that NASA follow best practices when it 

updated the 2011 analysis for the Northrop Grumman contract and 

ensure the analysis is updated as significant risks emerge in the future. 

NASA partially concurred with our recommendation and stated that the 

JWST program and project use a range of tools to assess all major 

contractors’ performance and that the project initiated a cost risk analysis 

of Northrop Grumman’s contract incorporating best practices and would 

update it when required by NASA policy.13 The status of NASA’s analysis 

and our evaluation of it are discussed later in the report.

The JWST project is currently on schedule with 8.75 months of schedule 

reserve remaining. However, all of JWST’s elements and major 

subsystems are within weeks of moving onto the project’s critical path, 

potentially reducing schedule reserve further. This is a tenuous position 

for the project given that it must complete five integration and test 

periods, three of which have not yet started. Testing can uncover 

problems that can be difficult or time-consuming to resolve, thereby 

adding schedule risk to the project and the unusual complexity of JWST 

further heightens these risks. To achieve mission success, the project will 

have to address over 100 technical risks and ensure that the project’s 

potential areas for mission failure are fully tested and understood before 

project launch in October 2018.

 12GAO, James Webb Space Telescope: Project Facing Increased Schedule Risk with 

Significant Work Remaining, GAO-15-100 (Washington, D.C.: Dec. 15, 2014).

13NASA classifies JWST as a single-project program—those which tend to have long 

development and operational lifetimes and represent a large investment. The JWST 

program office is based at NASA headquarters, and the project office is based at 

Goddard.

Previous GAO Reviews of 

JWST Project

JWST Is Meeting 

Schedule 

Commitment with 

Majority of Testing 

and Challenging 

Integration Work Still 

to Come
Page 12 GAO-16-112 James Webb Space Telescope

Overall project schedule reserve, currently at 8.75 months, remains 

above Goddard requirements and the project’s plan—which was set 

above the Goddard standard at the replan in 2011 and included more 

reserve than required. However, as shown in figure 4 below, the use of 

schedule reserve on any element or major subsystem—two of which have 

entered integration and testing phases—may reduce the overall project 

schedule reserve. While some use of schedule reserve is expected, the 

proximity of each element and major subsystem schedule to the critical 

path means that the project must prioritize the mitigations when problems 

occur.

Figure 4: Schedule Reserve Held by Each Element and Major Subsystem for the 

James Webb Space Telescope in 2014 and 2015

Project Is Above Schedule 

Reserve Standards with 

Significant Testing on the 

Horizon
Page 13 GAO-16-112 James Webb Space Telescope

Overall, the project has used more than 30 percent of its schedule 

reserve established at the time of the replan in 2011 to address technical 

challenges.14 Our prior work has shown that it is in integration and testing 

where problems are most likely to be found and as a result, schedules 

tend to slip. As we found in 2012, the project has a set amount of time 

allocated to the final three integration and test efforts over the next 3 

years, with between 2 and 4 months for each. This time could easily be 

used if a significant problem occurred.15 For example, the OTIS 

integration and test period—the first major integration involving OTE and 

ISIM—planned to start in 2016 currently has 3 months of schedule 

reserve allocated at the end of testing. The final event in the OTIS 

integration and test effort is a cryovacuum test that takes approximately 3 

months to complete. If an issue occurred that required stopping and 

repeating the cryovacuum test, this reserve could easily be exhausted. 

Additionally, as the project moves further into integration and testing, 

events become more serial so flexibility will be diminished. Issues 

uncovered in integration and testing also tend to be more expensive to 

mitigate, due to increased schedule pressure.

To prevent the use of additional schedule reserve, the project and its 

contractor for OTIS testing are taking proactive steps to reduce risk 

before testing needs to commence by ensuring the availability and 

readiness of test equipment and the cryogenic chamber to be used to test 

the optics of JWST. For example, the project’s contractor that is to test 

the optics has conducted two of three optical ground support equipment 

tests on a replica of the OTE with 2 of 18 primary mirror segments 

installed. According to the project, the first test met its intended objectives 

and provided valuable insight into the performance of the ground support 

equipment and preparation of the cryogenic chamber. The second test 

was completed in October 2015, and project officials are currently 

analyzing the results. The third test is to build upon these findings to 

provide further confidence for the eventual OTIS testing. Additionally, the 

contractor performed several risk mitigation activities, including additional 

testing of the large cryogenic chamber that will be used for OTIS testing, 

 14The 2011 baseline plan had 13 months of schedule reserve. However, by accelerating 

some work, the project was able to increase the schedule reserve to 14 months in June 

2012. 

15GAO, James Webb Space Telescope: Actions Needed to Improve Cost Estimate and 

Oversight of Test and Integration, GAO-13-4 (Washington, D.C.: Dec. 3, 2012).
Page 14 GAO-16-112 James Webb Space Telescope

which revealed several issues, including a leak in the cryogenic chamber 

that would have had major impacts if not discovered and repaired before 

OTIS testing began.

The project has used schedule reserve in 2015 to address various 

technical problems that have arisen. More specifically, the project 

experienced several problems with ISIM and OTE, elements in the two of 

five integration and test phases that have begun. For example, the ISIM 

heat straps—flexible straps that are to conduct energy and heat away 

from the instruments—did not perform as expected in testing. An 

investigation revealed design issues with the parts as delivered from the 

supplier. As a result, the heat straps were redesigned and reinstalled, 

which required the use of schedule reserve. Additionally, as a result of 

these and other issues, the beginning of the third cryovacuum test was

delayed by 3 months. ISIM currently holds 1.75 months in schedule 

reserve—down from 4.5 months as we found last year—from its overall 

schedule reserve of 8.75 months to address any issues that may arise 

during the third cryovacuum test and before OTIS testing begins.

Additionally, the OTE element used about 2 months of schedule reserve 

this past year due to workmanship issues related to the 76 cryogenic 

harnesses that connect to JWST’s mirrors. According to program and 

contractor officials, the majority of these harnesses were damaged due to 

use of inappropriate tooling by the supplier. The damage was not 

discovered until some of the harnesses were installed on the OTE. The 

harnesses were removed for inspection with most requiring repairs or 

replacement. According to contractor officials, initially, the harnesses 

would have been installed at Northrop Grumman’s facility in Redondo 

Beach, California, but due to the workmanship issues, and in an effort to 

preserve as much schedule as possible, all but two of the harnesses are 

being installed at Goddard after the OTE was transferred there to begin 

the integration of the mirrors with the backplane.

Various spacecraft challenges during the past year have used about 3

months of schedule reserve. For example, Northrop Grumman planned 

for certain integration activities to be conducted concurrently. However, 

according to project officials, due to safety and access to the spacecraft 

bus, the work had to be completed sequentially instead which took longer 

than expected. Additionally, a propellant tank required redesign and 

rework to meet its requirements. Spacecraft bus structure integration has 

been completed and the bus assembly recently completed various fit 

Schedule Reserve 

Continues to Be Used to 

Address Technical 

Challenges
Page 15 GAO-16-112 James Webb Space Telescope

checks and acoustics and dynamics testing in preparation for the 

spacecraft integration and testing phase to begin in 2016.

Schedule reserve for the sunshield was reduced to 9.25 months—two 

weeks from the critical path—due to various manufacturing challenges, 

and additional reserves will likely be needed in the near future. For 

example, an anomaly with the membrane retention devices—which need 

to operate correctly to ensure that the sunshield can unfold properly—

during qualification testing required a redesign of the parts. According to 

contractor officials, when the devices were released, the contact between 

the metal surfaces moving adjacent to one another resulted in a small 

amount of debris being generated. Project officials expressed concern 

that the debris posed a risk of damaging other parts of the telescope. A 

new design has since been tested and proven to no longer pose the 

same risk. Additionally, coordinating the testing of the five individual 

layers of the sunshield created some delays. The five layers of the 

sunshield are currently in various stages of assembly, with two layers

having been delivered to Northrop Grumman from the supplier in April 

2015 and November 2015, respectively. In addition, in October 2015, the 

project reported that a piece of flight hardware for the sunshield’s mid 

boom assembly was irreparably damaged during vacuum sealing in 

preparation for shipping. The effect of the accident on the schedule has 

not yet been determined as project and prime contractor officials are 

currently determining the path forward.

The cryocooler continued to experience technical challenges in 2015 that 

used schedule reserve and delayed its delivery. Although it has now been 

delivered—approximately 18 months later than planned—the cryocooler 

remains a schedule risk as it begins testing. Northrop Grumman delivered 

the compressor assembly—the third and final cryocooler component to be 

delivered after the cold head assembly and electronics assembly—to JPL 

in July 2015. Over the last several years, the project has accommodated 

a series of cryocooler schedule slips by reordering and compressing 

JPL’s test schedule and resequencing the spacecraft bus integration 

schedule. For example, several tests that were initially planned to be 

conducted on flight hardware will now be conducted on the spare 

hardware later into JWST’s integration and testing phase and closer to 

launch. Additionally, in May 2015, the project used 3 weeks of reserve 

and removed 3 weeks of lower priority and redundant items from the 

Cryocooler Delivered but 

Remains a Significant 

Schedule Risk during 

Testing
Page 16 GAO-16-112 James Webb Space Telescope

planned 40 weeks of acceptance and end-to-end testing.16 The project 

took these actions to accommodate a further delay in the delivery of the 

compressor assembly. According to contractor officials, the delay was 

primarily caused by the contractor not scheduling enough time to 

complete the bake out—a process whereby moisture is removed by 

heating the compressor and pumping helium through it. Table 1 below 

shows the tests removed from the acceptance and end-to-end testing.

Table 1: Tests Removed from Cryocooler Acceptance and End-to-End Testing 

Tests removed Duration Purpose

Redundant Thermal Performance tests 1 week Expose hardware to required thermal extremes and verify cooler 

subsystem functional and performance requirements

Electromagnetic Interference / Electromagnetic 

Compatibility opportunity test

1 week Provide an early indication of electro-magnetic interference and 

compatibility issues with the electronics

Low priority tests from End-to-End Phase 1 1 week Perform system level tests to determine thermal margins and 

nominal subsystem operating conditions and various risk 

reduction tests

Source: GAO analysis of National Aeronautics and Space Administration (NASA) data. | GAO-16-112

According to JPL officials, the thermal tests removed were from those that 

would have tested the cryocooler compressor assembly with the 

electronics assembly. A program official stated that the risk of eliminating 

these tests has been reduced now that the compressor and electronics 

assembly have been tested together for the first time. NASA and JPL 

assessed the removal of the electromagnetic interference and 

electromagnetic compatibility testing as low risk, and a program official 

stated that additional parallel activities and testing of the spare electronics 

to further mitigate the risk have been added. However, various integration 

and test experts we spoke with noted that eliminating testing is a sign that 

the project may be taking on additional risk because discovery of issues 

may be pushed to higher levels of testing or to later in the testing phase, 

where problems are more costly and time consuming to address. 

To accommodate any delays to testing or problems that may be found, 

JPL currently maintains 12 weeks of reserve for acceptance and end-toend testing of the cryocooler. According to JPL officials, a key driver in 

 16The primary objective of acceptance testing is to verify cryocooler subsystem-level 

requirements. End-to-end testing is to validate system-level performance and establish 

performance margins for the overall MIRI thermal system.
Page 17 GAO-16-112 James Webb Space Telescope

deciding to eliminate testing instead of using additional schedule reserve 

was to retain as much as possible in the event that a test has to be 

stopped and restarted—which would require approximately 5 weeks—in 

addition to the time it takes to mitigate a problem. At the completion of the 

acceptance and end-to-end testing programs, the cryocooler is needed 

for spacecraft integration and testing—when the spacecraft and sunshield 

are integrated—no later than August 2, 2016. Spacecraft integration and 

test is followed by the final observatory level integration and test—

completing the telescope—which is expected to begin in September 

2017. The cryocooler’s testing flow and schedule reserve leading to its 

integration with the spacecraft is depicted in figure 5 below.

Figure 5: Cryocooler Subsystem Integration and Test

Because the development and delivery of the cryocooler by Northrop 

Grumman took significantly longer than expected and to maintain the 12 

weeks of reserve, JPL must complete acceptance and end-to-end testing 

in a more schedule-compressed environment. However, challenges have 

persisted in bringing the cryocooler flight model to testing and completing 

development of the spare model which could be needed if the flight model 

is not available in time to be integrated into the observatory for launch. 

For example, despite having an extra 18 months to prepare for the 
Page 18 GAO-16-112 James Webb Space Telescope

cryocooler testing due to the delay in the delivery of the compressor 

assembly, the project noted concerns with JPL’s readiness to accept the 

flight hardware. Specifically, a procedure error led to an interruption of the 

ongoing testing of the electronics assembly and fit checks of the flight 

cooler tower assembly and flight refrigerant line deployable assembly 

were delayed because the procedures were late in being completed.

JWST is one of the most technologically complex projects NASA has 

undertaken. The project incorporates nine critical technologies—

technologies that are required for the project to successfully meet 

requirements—whereas we found the average technology development 

project at NASA incorporates an average of 2.3 critical technologies.17

JWST also incorporates 15 pre-existing technologies that are being 

leveraged from previous development efforts.

Future testing on JWST has to reduce a significant amount of risk before 

the October 2018 launch. The project identifies and maintains a list of 

risks—currently with 102 items—that need to be tested and mitigated to 

an acceptable level in the next 3 years. According to the project, 

approximately 25 of these risks are not likely to be closed until the 

conclusion of the observatory integration and test phase—just prior to 

project launch. This is the point where the project has determined that no 

further mitigations are feasible and that these risks have been tested per 

a plan to reduce the risk, when possible, to an acceptable level. In some 

cases, it may take years to resolve a particular risk. For example, the 

project continues to track a risk related to the release mechanisms that 

hold the spacecraft and the OTE together for launch. Once in space, they 

are to activate and release to allow the OTE to separate from the 

spacecraft. If the mechanisms do not operate correctly, mission failure will 

occur. This risk was identified in January 2014 at the spacecraft critical 

design review. During testing, these devices were causing excessive 

shock when performing their releasing function. After redesign, the project 

is continuing to work on resolving the underlying problem and qualifying 

the new design. Additionally, while testing the redesigned mechanism, a 

new concern arose that it could release early which would cause mission 

failure. According to a program official, the redesigned mechanism is not 

 17GAO, NASA: Assessments of Selected Large-Scale Projects, GAO-15-320SP 

(Washington, D.C.: Mar. 24, 2015).

Unusual Complexity of the 

JWST Project Heightens 

the Importance of 

Integration and Testing 

Phase to Reduce 

Substantial Risks
Page 19 GAO-16-112 James Webb Space Telescope

needed for spacecraft integration and test until summer of 2016 and 

therefore these issues do not pose a significant schedule concern at this 

time.

As integration and testing moves forward, the project will need to be able 

to resolve problems in a timely manner to stay on schedule. A backlog of 

unresolved problems and risks may indicate there may not be enough 

schedule left before launch to complete all necessary work. The project 

keeps track of these problems via problem reports and problem failure 

reports.18 Thus, ensuring that problem reports and problem failure reports 

are resolved in a timely manner is key to successfully launching the 

project on time. Project officials reported that they do not track problem 

report and problem failure report closure rates over time; instead, they 

monitor the reports to ensure that they are closed before subsequent test 

events and receive monthly briefings from the contractors on the status of 

their progress. According to project officials responsible for the ISIM and 

OTIS development and testing, while there are numerous problem reports 

open at any given time, they are comfortable with the number of open 

reports at this stage of the project. Northrop Grumman officials reported 

that development of the OTE, sunshield, and spacecraft are on track with 

respect to problem and failure reports, which they refer to as 

nonconformance reports. Additionally, experts we spoke with told us that 

addressing requests for action (RFAs) from project reviews is important 

because RFAs are written to identify potential risks to the project. Since 

the spacecraft critical design review in January 2014, the project has 

closed 14 RFAs from that review while one related to the release 

mechanism noted above remains open. The project tracks and reports 

open RFAs to senior management at NASA, and we will continue to 

examine the open RFA and additional RFAs that result from future 

reviews to monitor their timely closure.

The extent of JWST’s deployments—which are necessary because JWST 

must be stowed for launch to fit in the launch vehicle—means the 

telescope could fail to operate as planned in an extensive number of 

ways. According to project officials, there are over 100 different ways that 

a failure could occur, referred to as single point failure modes, across 

 18Problem reports are generated when any discrepancy between expected and actual 

results is detected and can be initiated by anyone involved in the flight hardware 

procedure. Problem failure reports are elevated from problem reports when the issue 

requires extensive investigation or repair.
Page 20 GAO-16-112 James Webb Space Telescope

hundreds of individual items in the observatory.19 Each of these could 

result in a loss of minimum mission objectives, and thus needs to be fully 

tested and understood. Nearly half of the single point failure modes 

involve the deployment of the sunshield. The approval of single point 

failures requires written justification from the project including sound 

engineering judgement, supporting risk analysis, and implementation of 

measures to mitigate the risk to acceptable levels. The project’s mission 

systems engineers have developed justifications and mitigation strategies 

for its single point failures, and project officials expect these to be 

summarized and submitted to the agency prior to launch. According to 

project officials, this approach is consistent with other high-priority NASA 

missions, which require the most stringent design and development 

approach that NASA takes to ensure the highest level of reliability and 

longevity on orbit.

The JWST project continued to meet its cost commitments throughout 

fiscal year 2015 despite cryocooler delays that used a disproportionate 

amount of cost reserves. However, the project required larger than 

planned workforce levels to complete new and existing work, which poses 

a cost threat in future years if levels do not decrease. To help manage the 

project and account for new risks since the 2011 replan, JWST project 

officials conducted a cost risk analysis of the Northrop Grumman contract. 

We found that while the cost risk analysis substantially met best 

practices, these officials do not plan to periodically update it. Instead, the 

project is using risk-adjusted analyses to update and inform its cost 

position. However, we found that this method is a simplified version of a 

cost-risk analysis that does not contain the same rigor or allow the project 

to prioritize risks. Furthermore, we found anomalies in the contractorprovided data rendering the results of the analyses unreliable. Finally, we 

also found the project lacks an independent surveillance mechanism for 

the data to ensure anomalies are corrected by the contractor before being 

incorporated into larger analyses. As a result, the project is relying 

partially on unreliable information to inform its cost and schedule 

decision-making.

 19A single point failure is an independent element of a system, the failure of which would 

result in loss of objectives, hardware, or crew.

Project Continues to 

Meet Cost 

Commitments but 

Contractor Workforce 

Increases and 

Unreliable Contractor 

Performance Data 

Pose Management 

Challenges
Page 21 GAO-16-112 James Webb Space Telescope

Project officials managed JWST within its allocated budget for the fourth 

consecutive year since the 2011 replan. Additionally, the project’s fiscal 

year 2016 budget request to Congress is consistent with its cost 

commitment. According to preliminary estimates, at the end of fiscal year 

2015, the project spent $68 million dollars more than planned at the 

beginning of the fiscal year, carrying over less money into fiscal year 

2016 than originally planned. As in past years, the project used a portion 

of its cost reserves to address technical challenges that included funding 

activities to address significant delays with the cryocooler. The project 

also used program-level cost reserves to pay for new work that included 

conducting additional thermal verification tests and risk reduction 

activities, such as an analysis to better understand how JWST will likely 

interact with its launch vehicle—the Ariane 5.

The cryocooler used a significant share of the project’s fiscal year 2015 

cost reserves—more than 50 percent—to fund the workforce for this effort 

and address technical issues. This is the fourth year in a row that the 

cryocooler used a substantive portion of the project’s cost reserves to 

further fund the subcontractor’s schedule delays in delivering its 

components. The project estimates that the overall cryocooler 

development cost will be nearly 250 percent higher than baselined at the 

2011 replan. The Northrop Grumman cryocooler team forecasts that a 

larger workforce is needed until at least February 2016 when the spare 

compressor assembly is currently scheduled to be delivered. JPL will 

maintain the majority of its workforce through the conclusion of spare 

cryocooler testing. After testing concludes, its workforce is projected to 

decrease by about 50 percent. Project cost reserves will likely continue to 

be needed to fund cryocooler development and testing costs until fiscal 

year 2017 when JPL testing of the spare compressor assembly is 

scheduled to conclude.

Project Continues to Stay 

on Budget and Spend 

within Cost Commitment 

Despite Significant 

Cryocooler Cost Growth
Page 22 GAO-16-112 James Webb Space Telescope

While the project remains on cost, contractor work is costing more to 

complete because a larger workforce than planned was needed for 

components beyond the cryocooler, including Northrop Grumman for the 

sunshield, spacecraft, and OTE, and Harris for OTIS testing and 

preparation.20 This need derives from work taking longer than planned to 

complete and additional work requested by NASA. For example, Northrop 

Grumman’s workforce projections for fiscal year 2015 predicted a peak in 

the workforce in November 2014. However, the actual workforce peaked 

in February 2015 and continued to remain above the projected peak until 

August 2015. While workforce numbers have declined somewhat since 

February, these increases largely remained in place through the end of 

the fiscal year. In its role as prime contractor, Northrop Grumman’s 

workforce stayed within its budget in fiscal year 2015. From January 

through July of 2015, its workforce was exactly at its funding threshold in 

order to conduct new work and address technical issues for its body of 

work. In addition, larger workforces contributed to additional contractor 

cost for two other development efforts—OTIS testing and the 

cryocooler—requiring the use of additional project cost reserves. 

Looking forward, the primary threat to JWST meeting its long-term cost 

commitment is the prime contractor, which must continue to control its 

costs and decrease its workforce. For the past 20 months, Northrop 

Grumman’s actual workforce exceeded its projections. Figure 6 below 

illustrates the difference between the workforce levels that Northrop 

Grumman projected at the beginning of fiscal years 2014 and 2015 and 

its actual workforce levels for those periods.21

 20Workforce level is discussed in terms of number of Full Time Equivalents (FTE) working 

on the project. FTE is defined by the Office of Management and Budget in Circular No. A11 (2015) as the total number of regular straight-time hours worked (i.e., not including 

overtime or holiday hours worked) by employees divided by the number of compensable 

hours applicable. For example, if the total number of compensable hours applicable is 40 

hours a week and an employee works 60 hours during that week, he or she accounts for 

1.5 FTE, as he or she worked 1.5 times. Therefore, while a workforce increase or 

decrease may mean a change in the number of employees, it does not have to. Instead it 

may mean that the FTE levels worked have either increased or decreased. 

21Northrop Grumman updates its workforce projections and provides that information to 

the JWST project on a monthly basis to reflect the forecasted estimate at completion and 

accounting for new opportunities and threats.

Larger Than Planned 

Workforce Poses a Threat 

to Meeting Cost 

Commitment
Page 23 GAO-16-112 James Webb Space Telescope

Figure 6: Northrop Grumman Workforce Growth in Fiscal Years 2014 and 2015, Projections versus Actuals at Beginning of 

Fiscal Year

Based on its projections at the beginning of the fiscal year, Northrop 

Grumman exceeded its total fiscal year 2014 workforce monthly 

projections by about 12 percent, and exceeded its projections for fiscal 
Page 24 GAO-16-112 James Webb Space Telescope

year 2015 by about 20 percent. On average, in fiscal year 2015, Northrop 

Grumman was 121 FTEs above its projections each month, and at the 

end of fiscal year 2015, it exceeded its monthly projection for September 

2015 by 235 FTEs. While actuals have remained above projections since 

the workforce levels peaked in February 2015, Northrop Grumman 

currently projects that its workforce will decline throughout fiscal year 

2016, with the exception of August 2016, when additional work is 

projected to be needed for integration and testing, among other areas. 

However, this was the projection for both fiscal years 2014 and 2015 and 

has yet to happen. For example, while Northrop Grumman expected to be 

ramping down by the end of fiscal year 2014, its projections at the start of 

fiscal year 2015 were approximately 55 percent higher than where 

workforce levels were projected for the end of fiscal year 2014. The 

primary drivers that have increased the cost and size of the workforce 

under the prime contract have been the development of the sunshield and 

spacecraft and additional work NASA has requested. Over 60 percent of 

the cost increases are attributed to addressing technical concerns such 

as sunshield alignment and verification work, mechanical design 

integration, and spacecraft mass reduction. Northrop Grumman has 

covered additional costs pertaining to technical issues through its 

management reserves, and has not needed project cost reserves in fiscal 

years 2014 and 2015. The remaining cost increases are attributable to 

new contract scope which has been funded by the JWST program. Some 

of this new scope included additional spacecraft simulators, as well as 

new thermal risk reduction testing to verify the final design changes made 

to the core of the telescope–the region between all the observatory 

elements.

Approximately 15 percent of work remains on Northrop Grumman’s 

contract and its management reserves exceed the recommended 

minimum amount that should be held at the contractor level—10 percent 

or more of the cost of work remaining on the project. Significant 

decreases in the workforce are planned to occur in fiscal year 2017 when 

final hardware delivery to observatory integration and test is scheduled to 

take place. To incentivize the contractor to lower its workforce, project 

officials evaluate workforce management as part of NASA’s appraisal of 

Northrop Grumman’s performance in its award fee determinations. The 

project also communicates frequently with the contractor including phone 

calls, face to face meetings twice a month, and quarterly in-person 

management meetings to discuss workforce planning, among other 

subjects. The project has communicated the need to reduce the 

workforce size, but since Northrop Grumman has operated within its 
Page 25 GAO-16-112 James Webb Space Telescope

budget in fiscal years 2014 and 2015, the award fee it has received has 

not been reduced as a result of workforce size issues.

The subcontractor for OTIS testing, Harris, needed additional funding to 

cover cost overruns and additional work. Project cost reserves were 

utilized to pay for this work to maintain schedule through a contract 

change in January 2015. Over 55 percent of the increase was made to 

address cost overruns that resulted from increasing workforce levels to 

maintain schedule. The rest of the contract increase covered new scope. 

As a result, Harris is anticipating more work than originally planned for 

fiscal years 2016 and 2017. Despite the contract increases, Harris’s 

management reserves are 2.5 percent as of August 2015—significantly 

below the 10 percent cost of work remaining that is considered to be 

healthy. With over 25 percent of work remaining on its contract, this low 

level of reserves means that any additional overruns will likely need to be 

covered by project-held reserves.

We found that NASA’s 2014 cost risk analysis on Northrop Grumman’s 

remaining work substantially met best practices. In December 2014, we 

recommended that project officials update the 2011 JWST cost risk 

analysis utilizing best practices, and to update it periodically as significant 

risks emerge.22 NASA partially concurred with our recommendation 

stating that the program and project use a range of tools to assess the 

performance of the project and conducted a one-time update to the cost 

risk analysis in 2014. We found that NASA’s updated cost risk analysis 

substantially met best practices. For example, it incorporated subject 

matter expert input to model cost and schedule uncertainties from the 

prime contractor’s threats and opportunities list—both of which are 

components of the best practice of modeling a probability distribution for 

each cost element’s uncertainty based on data availability, reliability and 

variability. See appendix I for a list of best practices that we used to 

evaluate cost risk and uncertainty. In addition, NASA included correlation 

between elements to account for different cost elements being affected by 

the same external factors—another best practice.

However, the cost risk analysis also had some weaknesses as a result of 

not fully following best practices. For the first best practice noted above 

 22GAO-15-100.

NASA’s JWST 2014 Cost 

Risk Analysis Substantially 

Met Best Practices
Page 26 GAO-16-112 James Webb Space Telescope

on modeling probability distribution, NASA relied on the contractor’s risk 

data without conducting corroborating interviews with contractor 

personnel to obtain insight into threats and opportunities not listed in 

contractor data. For the same best practice, the detailed schedule that 

reflected all of the work that needed to be done by Northrop Grumman 

that was used for the cost risk analysis had some activity sequencing 

logic issues. For example, we found instances where activities listed were 

not sequentially linked to one another. As a result, this called into 

question the calculation of the critical path during simulations as well as 

the ability of the schedule to dynamically respond to changes, which it 

must do thousands of times during the risk simulations. Moreover, the 

JWST project does not plan to periodically update its cost risk analysis 

even as additional risks have emerged. JWST officials stated that the cost 

risk analysis was a time intensive process to complete and that the 

program and project use various tools consistent with best practices to 

assess all major contractors’ performance. Nonetheless, best practices 

call for conducting periodic updates to a cost risk analysis as a project 

progresses even if it is not experiencing problems. Updating the cost risk 

analysis is also part of the best practice of implementing a risk 

management plan with the contractor which calls for identifying and 

analyzing risk, planning for risk mitigation, and continually tracking risks. 

An accurate cost risk analysis is particularly vital to JWST because about 

70 percent of the project cost reserves have been used to address 

concerns that were not anticipated as threats by the project’s budget 

system. Failure to update the cost risk analysis as we recommended in 

2014 limits stakeholder confidence that the cost risk analysis prepared in 

2014 accurately reflects the project’s current financial status. Given this 

uncertainty, it is important for the project to have reliable information for 

the risks that are known to inform decision making.

One of the tools that the project has started to use in place of updating 

the cost risk analysis is a monthly risk-adjusted analysis to provide insight 

into potential future cost growth. The monthly risk-adjusted analyses are 

based on contractor EVM data that incorporate known threats to provide 

an estimate at completion (EAC) that is updated monthly by NASA for 

each contractor. The results of these analyses may then be compared to 

the contractors’ estimates and project cost reserves to provide insight into 

the project’s ability to cover future increases. Monthly risk-adjusted 

analyses demonstrate a commitment by NASA to manage and project 

future costs.

Unreliable EVM Data Limit 

Project’s Ability to Reliably 

Forecast Future Cost 

Growth for Contractors
Page 27 GAO-16-112 James Webb Space Telescope

However, we found that the risk-adjusted analyses do not serve as an 

adequate substitute for an updated cost risk analysis because they are a 

simplified version of a cost risk analysis that does not allow the project to 

prioritize risks or assign confidence levels to meet key milestones in the 

schedule consistent with best practices for cost risk analyses. 

Additionally, based on our analysis of contractor EVM data over 17 

months, we found that some of the data used to conduct the analyses 

were unreliable. First, we found that both Northrop Grumman and Harris 

were reporting optimistic EACs at the time of our analysis that did not 

align with their historical EVM performance and fell outside the low end of 

our independent EAC range. Second, we found various anomalies in 

contractor EVM data for both the Northrop Grumman and Harris work that 

they had not identified throughout the 17-month period we examined. The 

anomalies included unexplained entries for negative values of work 

performed (meaning that work was unaccomplished or taken away rather 

than accomplished during the reporting period), work tasks performed but 

not scheduled, or actual costs incurred with no work performed. For 

Northrop Grumman, many were relatively small in value ranging from a 

few thousand to tens of thousands of dollars. These anomalies are 

problematic because they distort the EVM data, which affect the 

projection of realistic EACs. We found that these anomalies occurred 

consistently within the data over a 17-month period, which brings into 

question the reliability of the risk-adjusted EAC analysis built upon this 

information. NASA did not provide explanations into the anomalies for 

either contractor. While the contractors were able to provide explanations 

for the anomalies upon request, their explanations or corrections were not 

always documented within EVM records. Some of the reasons the 

contractors cited that were not in the EVM records included tasks 

completed later than planned, schedule recovered on behind schedule 

tasks, and replanning of customer-driven tasks. Finally, like the cost-risk 

analysis in 2014, the risk-adjusted EAC analysis does not include 

interviews with contractor officials to gain insight into risks which may not 

be present in the contractors’ threats and opportunities list. Without 

updating the cost risk analysis, reconciling and documenting data 

anomalies, and utilizing reliable data for the risk-adjusted EAC, the JWST 

project does not have a reliable method to assess its cost reserve status 

going forward. This means that some of the cost information the project 

officials use to inform their decision making may indicate they are in good 

shape when the reality might be otherwise, and as result, project 

management may not have a solid basis for decision making.

In discussions with the contractors, we found that the project also lacks 

an independent surveillance mechanism, such as the Defense Contract 
Page 28 GAO-16-112 James Webb Space Telescope

Management Agency, to monitor contractors’ EVM data—provided to the 

project each month from two of the contractors. Surveillance entails 

reviewing a contractor’s EVM system with the purpose of focusing on how 

well a contractor is using its EVM system to manage cost, schedule, and 

technical performance. However, the lack of surveillance and the data 

anomalies in EVM data are problems we previously identified across 

NASA’s portfolio of major spaceflight projects. We found in November 

2012 that 4 of NASA’s 10 major spaceflight projects we reviewed had 

established formal independent surveillance reviews.23 For the 6 projects 

that did not have formal independent surveillance in place, we found that 

each provided evidence that they instituted monthly EVM data reviews, 

which according to project officials, helped them to continually monitor 

cost and schedule performance. However, we found that the rigor of both 

the formal and informal surveillance reviews was questionable given the 

numerous EVM data anomalies we found in the monthly EVM data. As a 

result, we recommended that NASA improve the reliability of project EVM 

data by requiring projects to implement a formal surveillance program that 

ensured anomalies in contractor-delivered and in-house monthly earned 

value management reports were identified and explained, and report 

periodically to the center and mission directorate’s leadership on relevant 

trends in the number of unexplained anomalies. Citing resource 

constraints, NASA partially concurred with the recommendation and 

commented that it did not plan to implement a formal surveillance 

program, but agreed that the reliability and utility of the EVM data needed 

to be improved and noted several steps it planned to take to do so. We 

continue to believe that implementing this recommendation would be 

beneficial and prevent anomalies in EVM data from occurring that we 

have identified on the JWST project. Implementing surveillance of EVM 

contractor data is a best practice listed in the NASA Earned Value 

Management Implementation Handbook and GAO’s Cost Estimating and 

 23GAO-13-22.
Page 29 GAO-16-112 James Webb Space Telescope

Assessment Guide.24 With adequate surveillance in place, the anomalies 

we found in the EVM data could have been identified earlier and 

corrective action could have been directed to the contractors to explain 

the anomalies in the data. Without implementing proper surveillance, the 

project may be utilizing unreliable EVM data in its analyses to inform its 

cost and schedule decision making.

NASA has taken steps to provide independent oversight of the JWST 

project. Independent oversight of the JWST project has played and will 

likely continue to play an important role leading up to JWST’s launch in 

October 2018. Before the 2011 replan, two groups examined JWST to 

address underlying concerns with schedule and cost and made 

recommendations that NASA implemented. On an ongoing basis until 

launch, the Standing Review Board and the Independent Verification & 

Validation (IV&V) facility are to continue to oversee progress on hardware 

and software development, identify concerns, and assist the project to 

identify solutions to reduce risk and improve JWST’s likelihood of 

success.25

Various groups internal and external to NASA have conducted reviews, 

provided insights, and identified schedule efficiencies to inform and 

enhance the project’s approach to managing the development of JWST. 

Prior to the 2011 replan and because of concerns raised at the JWST 

mission critical design review held in the spring of 2010, the Test 

 24GAO’s Cost Estimating and Assessment Guide (GAO-09-3SP) states that effective 

surveillance ensures that the key elements of the EVM process are maintained over time 

and on subsequent applications. The two goals associated with the EVM system 

surveillance ensure that the contractor is following its own corporate processes and 

procedures and confirm that the contractor’s processes and procedures continue to satisfy 

the American National Standards Institute guidelines. The organization that conducts 

surveillance of an EVM system must have designated authority and accountability for 

EVM system surveillance to assess how well a contractor applies its EVM system relative 

to the American National Standards Institute guidelines. Surveillance organizations should 

be independent of the programs they assess and should have sufficient experience in 

EVM. These requirements apply to all surveillance organizations, whether internal or 

external to the agency, such as consultants.

25IV&V is a process conducted by a party independent of the development effort that 

provides an objective assessment of a project’s processes, products, and risks throughout 

its life cycle and helps ensure that program performance, schedule, and budget targets 

are met.

Independent 

Oversight Has Been 

Built into JWST’s 

Development Life 

Cycle to Improve the 

Likelihood of Mission 

Success

Independent Oversight 

Has Provided Insight to 

Reduce Risk
Page 30 GAO-16-112 James Webb Space Telescope

Assessment Team was formed to address those concerns. Convened by 

the Astrophysics division of the NASA Science Mission Directorate, the 

team included nine members and three NASA consultants with 

considerable experience in systems engineering, instrument 

development, system verification, modeling and testing, and other areas 

focused on reviewing plans for the ISIM and OTIS cryogenic testing. The 

team was primarily tasked to determine whether (1) the test plans in place 

at that time were sufficient to test the relevant observatory functions, (2) 

the key optical and thermal objectives were clearly identified, (3) the test 

plans themselves were properly scoped and prioritized, (4) any 

duplicative or unnecessary tests existed in the plans, and (5) the plans 

were overly ambitious or optimistic regarding hardware performance and 

analysis capabilities. Their insights and recommendations have helped to 

decrease programmatic cost and future growth as well as to find schedule 

efficiencies. For example, they recommended OTIS testing duration be 

reduced from 167 to 90 days while still verifying critical functions of the 

telescope. Also prior to the 2011 replan, the Chair of the Senate 

Subcommittee on Commerce, Justice, Science, and Related Agencies 

asked that NASA set up a panel to review the JWST project because of 

concerns about cost growth and schedule delays. In response, NASA 

convened the Independent Comprehensive Review Panel to provide an 

independent, integrated perspective and response with the goal of 

providing recommendations that would lead to a successful launch while 

minimizing cost. At the conclusion of its work in October 2010, the panel 

made 22 recommendations to NASA to increase oversight, improve 

communications, and assist with risk management and mitigation, among 

other recommendations. NASA implemented all of these 

recommendations. Both of these reports have informed our ongoing 

reviews of the JWST project as we have incorporated many of the 

concerns on cost estimates and cost reserves into our methodology and 

reporting on the health and status of JWST as it moves forward.

Another aspect of independent oversight that is a key element of NASA’s 

strategic framework for managing space flight projects are Standing 

Review Boards which consist of technical experts who do not actively 

work on a specific project or program. The mission of the boards is to 

provide NASA senior management with objective information to ensure 

there is appropriate program and project management oversight to 

increase the likelihood of mission success. The boards help to determine 

the adequacy of programs’ (1) management approach, (2) technical 

approach, (3) integrated cost and schedule estimates and funding 

strategy, and (4) risk management, among others. NASA’s Independent 

Independent Standing 

Review Board Provides 

Additional Expertise and 

Insight to Help Manage 

Project
Page 31 GAO-16-112 James Webb Space Telescope

Program Assessment Office and various NASA centers organize these 

boards and coordinate their involvement at different reviews.26 The 

boards are involved at various agency-level reviews with some members 

participating in lower-level reviews at NASA’s different centers, in monthly 

reviews held by the projects and program, or in special reviews on a 

specific topic or set of issues. Standing Review Boards may also make 

non-binding recommendations after life-cycle reviews, but do not have 

programmatic or technical authority over the programs or projects. The 

Standing Review Board Handbook describes three types of boards that 

may be formed to provide independent oversight of programs or projects. 

See table 2 below for the three types of Standing Review Boards.

Table 2: Standing Review Board Types at the National Aeronautics and Space Administration

Type of Standing Review Board

Civil Service Consensus Board–No 

Expert Support

Composition of the Board

This type of board is made up of civil servants with no consultants or outside experts. 

The Standing Review Board produces a briefing report with findings and 

recommendations as well as requests for action from individual members that is briefed 

by the chair of the Standing Review Board. This report represents the holistic view of the 

board but this type of board allows for a minority report that is included as well. 

Civil Service Consensus Board with 

Expert Support

This type of board is made up of civil servants who may use consultants who are nonboard members that support the Standing Review Board and may interact with 

projects/programs on behalf of Standing Review Board members in specific areas. Like 

the civil servant consensus board, the holistic view of the board is presented but a 

minority report is included as well.

Non-Consensus Mixed Board This type of board is made up of both civil servants and non-civil servants who may have 

relevant expertise in a specific area. The chair of the board, who may be either a civil 

servant or a non-civil servant, presents his/her opinion on the status of the project or 

program based on the inputs from all members. A minority report is not included.

Source: GAO presentation of the National Aeronautics and Space Administration’s (NASA) Standing Review Board Handbook.| GAO-16-112

NASA’s Standing Review Board Handbook states that a civil servant 

consensus with no expert support is the preferred structure within NASA 

because experience demonstrates that a consensus board leads to a 

more meaningful discussion of the review findings and recommendations, 

especially where dissenting opinions are discussed. A non-consensus 

 26In late 2015, NASA’s Associate Administrator announced that the Independent Program 

Assessment Office, and its umbrella organization, the Office of Evaluation, would be 

dissolved by the end of the calendar year. As a result of this change, Mission Directorates, 

in coordination with the executing centers, will be responsible for selecting Standing 

Review Board members with assistance from the Office of the Chief Engineer and Office 

of the Chief Financial Officer to enable programmatic expertise.
Page 32 GAO-16-112 James Webb Space Telescope

mixed board provides only the perspective of the chairman. In 2015, 1 of 

33 active Standing Review Boards was a civil service consensus board 

with no expert support, 15 were civil service consensus with consultant 

support, and 17 were non-consensus mixed boards. Although NASA 

guidance prefers civil servant consensus boards, NASA officials told us 

that they have found it challenging to staff boards exclusively with civil 

servants for a number of reasons including availability of staff, finding a 

person with the appropriate skill set, and independence reasons, among 

others.

JWST has had a number of changes occur on the boards overseeing the 

project for different reasons. JWST has had a Standing Review Board 

since 2006 when a special review was conducted. During that review and 

from 2008 to 2014, the board was a non-consensus board led by an 

outside expert chosen by NASA senior officials. The experts were civil 

servants as well as non-civil-servant experts. In May 2014, the chairman 

retired, a new chair was appointed the same year, and NASA senior 

officials changed the board to a consensus board with consultants. 

Independent Program Assessment Office officials told us that board types 

can change for numerous reasons, including when a project or program 

enters a different phase of development that may require different 

technical skills or if all of the convening authorities request it. As a result 

of the retirement of the chairman, most of the 2008-2014 Standing 

Review Board members who were not civil servants but who had 

overseen JWST for more than 6 years were replaced and 2 civil servants 

were carried over to the new board. Consultant support was added for 

schedule analysis and in one technical area to support launch vehicle 

integration because NASA has never launched a mission on an Ariane 5 

rocket as it plans to do for JWST. Before retiring in 2015, the previous 

Standing Review Board chairman expressed the importance of having 

representation from JPL as a member of the board to provide experience 

working on unmanned spacecraft projects—but a JPL member could not 

be added since JPL employees are not civil servants and can only be 

consultants to the board. With the appointment of a new chairman in 

October 2015, there have been additional membership changes to the 

board including the addition of a JPL consultant.
Page 33 GAO-16-112 James Webb Space Telescope

NASA’s IV&V facility—which independently examines software 

development—reviews mission critical software for most NASA programs 

and projects to achieve the highest levels of safety and cost-effectiveness 

by ensuring that developed software will perform as required. Experts at 

the facility work to uncover high-risk errors early in the development life 

cycle of software for many NASA programs and projects. IV&V is a 

process whereby organizations can reduce the risks inherent in system 

development and acquisition efforts by having a knowledgeable party who 

is independent of the developer to determine whether the system or 

product meets the users’ needs and fulfills its intended purpose. IV&V 

applies software engineering best practices to risk elements on safety 

critical and mission-critical software throughout the development life 

cycle. We have found IV&V to be a leading practice for federal agencies 

in managing their complex, large-scale, or high-risk acquisition of 

programs.27 Software development is a challenge we have found on 

many different acquisitions—some space-related—in government 

programs that has led to schedule delays and cost growth. Examples 

include the F-35 Joint Strike Fighter, the Aegis Modernized Weapon 

System, NASA’s Stratospheric Observatory for Infrared Astronomy, and 

Geostationary Weather Satellite development, among others.28

The goal of IV&V is to examine the three following questions regarding 

software:

• Will the system do what it is supposed to do?

• Will the system not do what it is not supposed to do?

• Will the system perform as expected under adverse conditions?

IV&V is required to examine software on all projects with a life cycle cost 

over $1 billion, other projects over $250 million with a higher risk 

classification, or those specifically selected by the NASA Chief, Safety 

and Mission Assurance. Once selected, a portfolio-based risk 

 27GAO, Information Technology: DHS Needs to Improve Its Independent Acquisition 

Reviews, GAO-11-581, (Washington, D.C.: Jul. 28, 2011).

28GAO, Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but 

Progress Still Lags, GAO-11-325, (Washington, D.C.: Apr. 7, 2011); GAO, Missile 

Defense: Mixed Progress in Achieving Acquisition Goals and Improving Accountability, 

GAO-14-351, (Washington, D.C.: Apr. 1, 2014); GAO, NASA: Assessments of Selected 

Large-Scale Projects, GAO-13-276SP, (Washington, D.C.: Apr. 17, 2013);GAO, 

Geostationary Weather Satellites: Launch Date Nears, but Remaining Schedule Risks 

Need to be Addressed, GAO-15-60, (Washington, D.C.: Dec. 16, 2014). 

Independent Oversight of 

Critical Software Confirms 

Development Is on 

Schedule with Concerns 

Regarding Large Number 

of Developers
Page 34 GAO-16-112 James Webb Space Telescope

assessment is developed to identify top-level mission capabilities and a 

risk based assessment process identifies the most important system 

capabilities and the software components that play the most important 

role in the mission. IV&V officials noted that due to limited resources, they 

examine mission and safety critical software and they do not have the 

funding to examine all programs or projects across NASA’s portfolio.

Generally, IV&V officials stated that they believe JWST’s software 

development is going well, but the testing that lies ahead—when the 

different components are integrated—will be a challenge. For example, 

IV&V officials noted that JWST’s software build is the largest they have 

reviewed for a science mission, but not the largest they have reviewed 

across NASA as some Human Exploration Operations are larger. They 

said that most of JWST’s software required to position and deploy the 

telescope will be examined by IV&V. However, they noted that JWST’s 

integration is more challenging, primarily due to the number of software 

developers involved. While most science programs or projects have two 

to four software developers, JWST has eight. JWST’s software 

development has been examined by IV&V since fiscal year 2004 and, 

according to officials, will likely continue to be examined until after launch 

when operations begin. IV&V officials said they do not examine the 

software for the launch vehicle.

While IV&V’s function requires independence from programs and 

projects, there have been recent changes in funding that have reduced its 

financial independence to some extent. Organizationally, the IV&V Facility 

remains independent by reporting to the Office of the Director of Goddard 

and the Office of Safety and Mission Assurance at NASA Headquarters—

not to the programs or projects it examines. However, financially, starting 

in 2015, an IV&V financial management official said that 75 percent of the 

IV&V’s funding came from NASA Headquarters via the Agency 

Management Operations fund and the remaining 25 percent was divided 

amongst the various mission directorates. This changed from the past 10 

years, when 100 percent of the IV&V Facility’s budget came from the 

Agency Management Operations to ensure the independence of the IV&V 

office. We have previously found that financial independence requires 

that the funding for IV&V be controlled by an organization separate from 

the development organization. This ensures that the effort will not be 

curtailed by having its funding diverted to other program needs, and that 
Page 35 GAO-16-112 James Webb Space Telescope

financial pressures cannot be used to influence the effort.29 As a user of 

IV&V’s expertise, JWST, via the use of program cost reserves, 

contributed a small portion of funding to the software IV&V facility to help 

fund their budget in fiscal year 2015. While this financial situation was 

new in fiscal year 2015, we will continue to monitor how NASA deals with 

funding the IV&V facility in the future to protect its independence.

The JWST project has made progress building, integrating, and testing 

significant portions of JWST in the past year within the commitments 

made at the time of the 2011 replan for both schedule and cost. With the 

third major integration and test period starting in 2016, resolving technical 

challenges in a timely manner, and ensuring the OTIS test goes smoothly 

are key to continued progress within the project’s schedule commitment. 

Additionally, reducing the size of Northrop Grumman’s workforce and 

controlling costs within the fiscal year 2016 budget will be key metrics to 

monitor to demonstrate the project can meet requirements within its cost 

commitment. In the past, the project has benefited from independent 

expertise, information, and recommendations to improve the 

management of the project.

Moving forward, the project may benefit from having more reliable data 

provided from its contractors to ensure that its EACs, which take into 

account risks and threats, are better able to inform its cost status. While 

the contractors were able to explain the anomalies, most had not been 

previously identified or documented. NASA used the data for its analyses, 

which subsequently raised questions about the reliability of those 

analyses. Making management decisions using unreliable data can result 

in bad decision making and can misinform the project on its long-term 

financial position which may have significant consequences if not 

corrected. We recommended in our December 2014 report that NASA 

conduct a cost risk analysis and follow best practices, which include 

updating it as risks change during the life of the program. Because the 

project is not going to conduct another cost risk analysis, putting 

independent surveillance in place to improve the accuracy of its riskadjusted analysis—despite its weaknesses relative to the information a 

cost risk analysis provides—will provide better information to inform its 

decision making. In November 2012, we recommended that NASA 

 29GAO-11-581.

Conclusions
Page 36 GAO-16-112 James Webb Space Telescope

improve the reliability of project EVM data by requiring its major 

spaceflight projects to implement a formal surveillance program that 

ensured anomalies in contractor-delivered data and in-house monthly 

EVM reports were identified and explained. NASA partially concurred with 

this recommendation but has not taken steps to require surveillance on 

projects like JWST. However, we continue to believe that improving the 

surveillance on projects will help reduce data anomalies from occurring 

like the ones we identified on JWST, resulting in better information and 

analyses to inform project decision making.

To resolve contractor data reliability issues and ensure that the project 

obtains reliable data to inform its analyses and overall cost position, we 

recommend that the NASA Administrator direct JWST project officials to 

require the contractors to identify, explain, and document all anomalies in 

contractor-delivered monthly earned value management reports.

We provided a draft of this report to NASA for comment. In written 

comments, NASA agreed with our recommendation. These comments 

are reprinted in appendix IV. NASA also provided technical comments, 

which have been addressed in the report, as appropriate.

We are sending copies of the report to NASA’s Administrator and 

interested congressional committees. In addition, the report will be 

available at no charge on GAO’s website at http://www.gao.gov.

Should you or your staff have any questions on matters discussed in this 

report, please contact me at (202) 512-4841 or chaplainc@gao.gov. 

Contact points for our Offices of Congressional Relations and Public 

Affairs may be found on the last page of this report. GAO staff who made 

major contributions to this report are listed in appendix V.

Cristina T. Chaplain

Director

Acquisition and Sourcing Management

Recommendation for 

Executive Action

Agency Comments 

and Our Evaluation
Page 37 GAO-16-112 James Webb Space Telescope

List of Committees

The Honorable Richard C. Shelby

Chairman

The Honorable Barbara Mikulski

Ranking Member

Subcommittee on Commerce, Justice, Science, and Related Agencies

Committee on Appropriations

United States Senate

The Honorable John Culberson

Chairman

The Honorable Mike Honda

Acting Ranking Member

Subcommittee on Commerce, Justice, Science, and Related Agencies

Committee on Appropriations

House of Representatives

The Honorable Lamar Smith

Chairman

The Honorable Eddie Bernice Johnson

Ranking Member

Committee on Science, Space, and Technology

House of Representatives
Appendix I: Objectives, Scope, and 

Methodology

Page 38 GAO-16-112 James Webb Space Telescope

Our objectives were to assess (1) the extent to which technical 

challenges have impacted the James Webb Space Telescope (JWST) 

project’s ability to meet its schedule commitments, (2) the current cost 

status of the JWST project and the primary challenges that may influence 

the project’s ability to meet its future cost commitments, and (3) the extent 

to which independent oversight provides insight about project risks to 

management.

To assess the extent to which technical challenges have impacted the 

JWST project’s ability to meet its schedule commitments, we reviewed 

project and contractor schedule documentation, and held interviews with 

program, project, and contractor officials on the progress made and 

challenges faced building the different components of the telescope. We 

examined and analyzed monthly JWST project status reports to 

management to monitor schedule reserve levels and usage and potential 

risks and technical challenges that may impact the project’s schedule, 

and to gain insights on the project’s progress since our last report in 

December 2014. Further, we attended flight program reviews at the 

National Aeronautics and Space Administration (NASA) headquarters on 

a quarterly basis where the current status of the program was briefed to 

NASA headquarters officials outside of the project. We examined 

selected individual risks for elements and major subsystems from monthly 

risk registers prepared by the project to understand the likelihood of 

occurrence and impacts to the schedule based on steps the project is 

taking to mitigate the risks. We examined previous and current test 

schedules and plans to understand the sequence, what risks will be 

mitigated, which risks will continue, and any reductions to planned testing. 

Furthermore, we interviewed experts within and outside of NASA to 

identify criteria, best practices, and metrics that may be used to assess 

the project’s progress in reducing risk or provide insight into the health of 

the project. Finally, we interviewed project officials at Goddard, contractor 

officials from the Harris Corporation, the Jet Propulsion Laboratory, the 

Space Telescope Science Institute, and different divisions of Northrop 

Grumman Aerospace Systems concerning technological challenges that 

have had an impact on schedule, and the project’s and contractor’s plans 

to address these challenges.

To assess the current cost status of the JWST project and the primary 

challenges that may influence the project’s ability to meet its future cost 

commitments, we reviewed and analyzed program, project, contractor, 

and subcontractor data and documentation and held interviews with 

officials from these organizations. We reviewed JWST project status 

reports on cost issues to determine the risks that could impact cost. We 

Appendix I: Objectives, Scope, and 

Methodology
Appendix I: Objectives, Scope, and 

Methodology

Page 39 GAO-16-112 James Webb Space Telescope

analyzed contractor and subcontractor’s workforce plans against 

workforce actuals to determine whether contractors’ are meeting their 

workforce plans. We monitored and analyzed the status of program, and 

project cost reserves in current and future fiscal years to determine the 

project’s financial posture. We evaluated the cost risk analysis conducted 

by NASA of the remaining Northrop Grumman work to determine the 

extent to which all applicable best practices from GAO’s Cost Estimating 

and Assessment Guide were used to build the analysis.

1 Those best 

practices included the following:

• A probability distribution modeled each cost element’s uncertainty 

based on data availability, reliability, and variability.

• The correlation between cost elements was accounted for to capture 

risk.

• A Monte Carlo simulation model was used to develop a distribution of 

total possible costs and an S curve showing alternative cost estimate 

probabilities.

• The probability associated with the point estimate was identified.

• Contingency reserves were recommended for achieving the desired 

confidence level.

• The risk-adjusted cost estimate was allocated, phased, and converted 

to then year dollars for budgeting, and high-risk elements were 

identified to mitigate risks.

• A risk management plan was implemented jointly with the contractor 

to identify and analyze risk, plan for risk mitigation, and continually 

track risks.

We examined and analyzed earned value management (EVM) data from 

two of the project’s contractors to identify trends in performance, whether 

tasks were completed as planned, and likely estimates at completion. We 

also conducted analysis to ensure the reliability of the data over a 17-

 1

GAO, GAO Cost Estimating and Assessment Guide, GAO-09-3SP (Washington, D.C.: 

Mar. 2, 2009). The methodology outlined in the guide is a compilation of best practices 

that federal cost-estimating organizations and industry use to develop and maintain 

reliable cost estimates throughout the life of a government acquisition program.
Appendix I: Objectives, Scope, and 

Methodology

Page 40 GAO-16-112 James Webb Space Telescope

month period. In addition, we examined and analyzed risk-adjusted 

analyses from NASA to determine what information they provide to the 

project, the risks incorporated, their reliability, and how the project is 

utilizing this information. We also discussed our assessment of the 

project’s data and analysis with program and project officials to obtain 

their input.

To assess the extent to which independent oversight provides insight 

about project risks to management, we reviewed documentation and data 

from NASA relevant groups, the program, the project, and the Standing 

Review Board and held interviews with experts as well as officials from 

independent oversight entities. We analyzed NASA policy and guidance 

documents to understand the elements for setting up and managing a 

Standing Review Board. We also reviewed the Test Assessment Team 

and Independent Comprehensive Review Panel team reports to 

determine how independent oversight has provided insight to JWST in the 

past. We interviewed officials at NASA’s Independent Program 

Assessment Office, as well as past and current Standing Review Board 

members, to understand how Standing Review Boards are created, 

members are selected, and how structural and personnel changes are 

made over the life of NASA programs and projects, including JWST. We 

also interviewed and reviewed documentation and analysis provided by 

NASA’s Independent Verification and Validation group working on 

JWST’s software development to determine the extent to which this group 

is providing oversight of JWST software development, to determine the 

health of software development on JWST, and determine what kinds of 

problems remain. We did not independently review JWST’s software 

development.

Our work was performed primarily at NASA headquarters in Washington, 

D.C.; Goddard Space Flight Center in Greenbelt, Maryland; the 

Independent Verification and Validation facility in Fairmont, West Virginia; 

and by video teleconference with officials from the Independent Program 

Assessment Office at Langley Research Center, Hampton, Virginia. We 

also visited the Jet Propulsion Laboratory in Pasadena, California; 

Northrop Grumman Aerospace Systems in Redondo Beach, California; 

and the Space Telescope Science Institute in Baltimore, Maryland.

We conducted this performance audit from February 2015 to December 

2015 in accordance with generally accepted government auditing 

standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objectives. We believe 
Appendix I: Objectives, Scope, and 

Methodology

Page 41 GAO-16-112 James Webb Space Telescope

that the evidence obtained provides a reasonable basis for our findings 

and conclusions based on our audit objectives.
Appendix II: Elements and Major Subsystems 

of the James Webb Space Telescope (JWST) 

Observatory

Page 42 GAO-16-112 James Webb Space Telescope

Appendix II: Elements and Major 

Subsystems of the James Webb Space 

Telescope (JWST) Observatory
Appendix III: Organizational Chart for the 

James Webb Space Telescope (JWST) 

Program

Page 43 GAO-16-112 James Webb Space Telescope

a

The Jet Propulsion Laboratory is the contractor for the development of the cryocooler, but has 

subcontracted most of the work to a different division of Northrop Grumman than the one that is 

responsible for OTE, spacecraft, and sunshield development.

Appendix III: Organizational Chart for the 

James Webb Space Telescope (JWST) 

Program
Appendix IV: Comments from the National 

Aeronautics and Space Administration

Page 44 GAO-16-112 James Webb Space Telescope

Appendix IV: Comments from the National 

Aeronautics and Space Administration
Appendix IV: Comments from the National 

Aeronautics and Space Administration

Page 45 GAO-16-112 James Webb Space Telescope
Appendix V: GAO Contact and Staff 

Acknowledgements

Page 46 GAO-16-112 James Webb Space Telescope

Cristina Chaplain, (202) 512-4841 or chaplainc@gao.gov.

In addition to the contact named above, Shelby S. Oakley, Acting 

Director; Arthur Gallegos, Assistant Director; Jay Tallon; Assistant 

Director; Karen Richey, Assistant Director; Jason Lee, Assistant Director; 

Brian Bothwell; Patrick Breiding; Aaron Gluck; Laura Greifner; Michael 

Kaeser; Katherine Lenane; Silvia Porres; Carrie Rogers; Sylvia Schatz; 

and Ozzy Trevino made key contributions to this report.

Appendix V: GAO Contact and Staff 

Acknowledgements

GAO Contact

Staff 

Acknowledgments

(100065)
The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance 

to help Congress make informed oversight, policy, and funding decisions. 

GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no 

cost is through GAO’s website (http://www.gao.gov). Each weekday 

afternoon, GAO posts on its website newly released reports, testimony, 

and correspondence. To have GAO e-mail you a list of newly posted 

products, go to http://www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s website, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.

Subscribe to our RSS Feeds or E-mail Updates. 

Listen to our Podcasts and read The Watchblog.

Visit GAO on the web at www.gao.gov.

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-

4400, U.S. Government Accountability Office, 441 G Street NW, Room 

7125, Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations

Public Affairs

Please Print on Recycled Paper.