James Webb Space Telescope: Project on Track but May Benefit from Improved Contractor Data to Better Understand Costs
Released: 2015-12-17T00:00:00

Letter

Background

JWST is envisioned to be a large deployable, infrared-optimized space telescope and the scientific successor to the aging Hubble Space Telescope. JWST is being designed for a 5-year mission to find the first stars and trace the evolution of galaxies from their beginning to their current formation, and is intended to operate in an orbit approximately 1.5 million kilometers—or 1 million miles—from the Earth. With a 6.5-meter primary mirror, JWST is expected to operate at about 100 times the sensitivity of the Hubble Space Telescope. JWST’s science instruments are to observe very faint infrared sources and as such are required to operate at extremely cold temperatures. To help keep these instruments cold, a multi-layered tennis-court-sized sunshield is being developed to protect the mirrors and instruments from the sun’s heat. The sunshield and primary mirror are designed to fold and stow for launch and fit within the launch vehicle. When complete, the observatory segment of JWST is to include several elements (Optical Telescope Element (OTE), Integrated Science Instrument Module (ISIM), and spacecraft) and major subsystems (sunshield and cryocooler). The JWST project is divided into three major segments: the observatory segment, the ground segment, and the launch segment. The hardware configuration created when the Optical Telescope Element and the Integrated Science Instrument Module are integrated, referred to as OTIS, is not considered an element by NASA, but we categorize it as such for ease of discussion. Additionally, JWST is dependent on software to deploy and control various components of the telescope as well as collect and transmit data back to Earth. The elements, major subsystems, and software are being developed through a mixture of NASA, contractor, and international partner efforts. See figure 1 below for an interactive graphic that depicts the elements and major subsystems of JWST. For more information on JWST’s organizational structure, see appendix III.

Given JWST’s complexity, integration and test activities are comprised of five separate periods—two of which have already started—over the course of almost 7 years to build the observatory. During the test periods, the project works to mitigate risks to an acceptable level prior to launch. According to project officials, while some risks may be eliminated entirely through various mitigation strategies, others will be accepted as residual risks that remain upon launch. See figure 2 below for the overall planned integration and test flow for JWST that includes the remaining schedule reserve—or extra time built into the schedule to address any issues found.

For the majority of the work remaining, the JWST project will rely on three contractors: Northrop Grumman, Harris (formerly Exelis), and the Space Telescope Science Institute (STScI). Northrop Grumman plays the largest role, developing the sunshield, the OTE, the spacecraft, and a cooling subsystem for the Mid-Infrared Instrument (MIRI). Northrop Grumman performs most of this work under a prime contract with NASA, but its work on the MIRI cooler is performed under a separate subcontract with the Jet Propulsion Laboratory (JPL). Harris is manufacturing the test equipment, equipping the test chamber, and assisting in the testing of the optics of JWST. Finally, STScI will collect and evaluate research proposals from the scientific community and will receive and store the scientific data collected, both of which are services that they currently provide for the Hubble Space Telescope. Additionally, STScI is responsible for developing the ground system that manages and controls the telescope’s observations on behalf of NASA.

The MIRI instrument, one of the four instruments within ISIM, requires a dedicated, interdependent two-stage cooler subsystem designed to cool the infrared light detector to about 6.7 Kelvin (K), just above absolute zero. This cooler is referred to as a cryocooler and works by moving helium gas through about 10 meters (approximately 33 feet) of refrigerant lines located on the sun-facing surface of the JWST observatory to the colder, shaded side where the ISIM is located. According to NASA officials, a cryocooler of this configuration has never been developed or flown in space before. See figure 3 below for an illustration of the MIRI cryocooler on JWST and the varying temperatures needed in different areas of the telescope.

Cost and Schedule Reserves for NASA Projects

Complex development efforts like JWST face myriad risks and unforeseen technical challenges which oftentimes can become apparent during integration and testing. To accommodate these risks and unknowns, projects reserve extra time in their schedules—which is referred to as schedule reserve—and extra money in their budgets— which is referred to as cost reserve. Schedule reserve is allocated to specific activities, elements, and major subsystems in the event there are delays or to address unforeseen risks. Each JWST element and major subsystem has been allocated schedule reserve. When an element or major subsystem exhausts schedule reserve, it may begin to affect schedule reserve on other elements or major subsystems whose progress is dependent on prior work being finished for its activities to proceed. The element or major subsystem with the least amount of schedule reserve determines the critical path for the project. Any delay to an activity that is on the critical path will reduce schedule reserve for the whole project, and could ultimately impact the overall project schedule.

Cost reserves are additional funds within the project manager’s budget that can be used to address unanticipated issues for any element or major subsystem and are used to mitigate issues during the development of a project. For example, cost reserves can be used to buy additional materials to replace a component or, if a project needs to preserve schedule reserve, reserves can be used to accelerate work by adding shifts to expedite manufacturing and save time. NASA’s Goddard Space Flight Center (Goddard)—the NASA center with responsibility for managing JWST—has issued procedural requirements that establish the levels of both cost and schedule reserves that projects must hold at project confirmation. After this point, a specified amount of schedule reserve continues to be required throughout the remainder of development. In addition to cost reserves held by the project manager, management reserves are funds held by the contractors that allow them to address cost increases throughout development. We have found that management reserves should contain 10 percent or more on the cost to complete a project and are used to address different issues.

History of Cost Growth, Low Project Reserves, and Schedule Delays

JWST has experienced significant increases to project costs and schedule delays. Prior to being approved for development, cost estimates of the project ranged from $1 billion to $3.5 billion with expected launch dates ranging from 2007 to 2011. Before 2011, early technical and management challenges, contractor performance issues, low level cost reserves, and poorly phased funding levels caused JWST to delay work after confirmation, which contributed to significant cost and schedule overruns, including launch delays. The Chair of the Senate Subcommittee on Commerce, Justice, Science, and Related Agencies requested from NASA an independent review of JWST in June 2010. In response, NASA commissioned the Independent Comprehensive Review Panel, which issued its report in October 2010, and concluded that JWST was executing well from a technical standpoint, but that the baseline funding did not reflect the most probable cost with adequate reserves in each year of project execution, resulting in an unexecutable project. Following this review, the JWST program underwent a replan in September 2011, and Congress in November 2011 placed an $8 billion cap on the formulation and development costs for the project. On the basis of the replan, NASA rebaselined JWST with a life-cycle cost estimate of $8.835 billion that included additional money for operations and a planned launch in October 2018. The revised life-cycle cost estimate included a total of 13 months of funded schedule reserve. In the President’s fiscal year 2013 budget request, NASA reported a 66 percent joint cost and schedule confidence level—lower than the 70 percent level noted in NASA procedural requirements—for these cost and schedule baselines. A joint cost and schedule confidence level is the process NASA uses to assign a percentage to the probable success of meeting cost and schedule estimates and is part of the project’s estimating process.

Previous GAO Reviews of JWST Project

In December 2014, we found that the project was progressing within the 2011 replan for both cost and schedule. We reported on technical challenges with JWST elements and major subsystems that had consumed a portion of the cost and schedule reserves. We also found that the cryocooler remained an ongoing challenge and continued to use a disproportionate amount of cost reserves. Finally, we found that NASA had not conducted a cost risk analysis since the 2011 replan. A cost risk analysis determines the reliability of a program’s cost estimate by determining a program’s cost drivers and the risk of cost overruns through an analysis that links historical schedule information along with technical issues and uncertainties in schedule and cost. Since new risks had emerged, we recommended that NASA follow best practices when it updated the 2011 analysis for the Northrop Grumman contract and ensure the analysis is updated as significant risks emerge in the future. NASA partially concurred with our recommendation and stated that the JWST program and project use a range of tools to assess all major contractors’ performance and that the project initiated a cost risk analysis of Northrop Grumman’s contract incorporating best practices and would update it when required by NASA policy. The status of NASA’s analysis and our evaluation of it are discussed later in the report.

JWST Is Meeting Schedule Commitment with Majority of Testing and Challenging Integration Work Still to Come

The JWST project is currently on schedule with 8.75 months of schedule reserve remaining. However, all of JWST’s elements and major subsystems are within weeks of moving onto the project’s critical path, potentially reducing schedule reserve further. This is a tenuous position for the project given that it must complete five integration and test periods, three of which have not yet started. Testing can uncover problems that can be difficult or time-consuming to resolve, thereby adding schedule risk to the project and the unusual complexity of JWST further heightens these risks. To achieve mission success, the project will have to address over 100 technical risks and ensure that the project’s potential areas for mission failure are fully tested and understood before project launch in October 2018.

Project Is Above Schedule Reserve Standards with Significant Testing on the Horizon

Overall project schedule reserve, currently at 8.75 months, remains above Goddard requirements and the project’s plan—which was set above the Goddard standard at the replan in 2011 and included more reserve than required. However, as shown in figure 4 below, the use of schedule reserve on any element or major subsystem—two of which have entered integration and testing phases—may reduce the overall project schedule reserve. While some use of schedule reserve is expected, the proximity of each element and major subsystem schedule to the critical path means that the project must prioritize the mitigations when problems occur.

Overall, the project has used more than 30 percent of its schedule reserve established at the time of the replan in 2011 to address technical challenges. Our prior work has shown that it is in integration and testing where problems are most likely to be found and as a result, schedules tend to slip. As we found in 2012, the project has a set amount of time allocated to the final three integration and test efforts over the next 3 years, with between 2 and 4 months for each. This time could easily be used if a significant problem occurred. For example, the OTIS integration and test period—the first major integration involving OTE and ISIM—planned to start in 2016 currently has 3 months of schedule reserve allocated at the end of testing. The final event in the OTIS integration and test effort is a cryovacuum test that takes approximately 3 months to complete. If an issue occurred that required stopping and repeating the cryovacuum test, this reserve could easily be exhausted. Additionally, as the project moves further into integration and testing, events become more serial so flexibility will be diminished. Issues uncovered in integration and testing also tend to be more expensive to mitigate, due to increased schedule pressure.

To prevent the use of additional schedule reserve, the project and its contractor for OTIS testing are taking proactive steps to reduce risk before testing needs to commence by ensuring the availability and readiness of test equipment and the cryogenic chamber to be used to test the optics of JWST. For example, the project’s contractor that is to test the optics has conducted two of three optical ground support equipment tests on a replica of the OTE with 2 of 18 primary mirror segments installed. According to the project, the first test met its intended objectives and provided valuable insight into the performance of the ground support equipment and preparation of the cryogenic chamber. The second test was completed in October 2015, and project officials are currently analyzing the results. The third test is to build upon these findings to provide further confidence for the eventual OTIS testing. Additionally, the contractor performed several risk mitigation activities, including additional testing of the large cryogenic chamber that will be used for OTIS testing, which revealed several issues, including a leak in the cryogenic chamber that would have had major impacts if not discovered and repaired before OTIS testing began.

Schedule Reserve Continues to Be Used to Address Technical Challenges

The project has used schedule reserve in 2015 to address various technical problems that have arisen. More specifically, the project experienced several problems with ISIM and OTE, elements in the two of five integration and test phases that have begun. For example, the ISIM heat straps—flexible straps that are to conduct energy and heat away from the instruments—did not perform as expected in testing. An investigation revealed design issues with the parts as delivered from the supplier. As a result, the heat straps were redesigned and reinstalled, which required the use of schedule reserve. Additionally, as a result of these and other issues, the beginning of the third cryovacuum test was delayed by 3 months. ISIM currently holds 1.75 months in schedule reserve—down from 4.5 months as we found last year—from its overall schedule reserve of 8.75 months to address any issues that may arise during the third cryovacuum test and before OTIS testing begins.

Additionally, the OTE element used about 2 months of schedule reserve this past year due to workmanship issues related to the 76 cryogenic harnesses that connect to JWST’s mirrors. According to program and contractor officials, the majority of these harnesses were damaged due to use of inappropriate tooling by the supplier. The damage was not discovered until some of the harnesses were installed on the OTE. The harnesses were removed for inspection with most requiring repairs or replacement. According to contractor officials, initially, the harnesses would have been installed at Northrop Grumman’s facility in Redondo Beach, California, but due to the workmanship issues, and in an effort to preserve as much schedule as possible, all but two of the harnesses are being installed at Goddard after the OTE was transferred there to begin the integration of the mirrors with the backplane.

Various spacecraft challenges during the past year have used about 3 months of schedule reserve. For example, Northrop Grumman planned for certain integration activities to be conducted concurrently. However, according to project officials, due to safety and access to the spacecraft bus, the work had to be completed sequentially instead which took longer than expected. Additionally, a propellant tank required redesign and rework to meet its requirements. Spacecraft bus structure integration has been completed and the bus assembly recently completed various fit checks and acoustics and dynamics testing in preparation for the spacecraft integration and testing phase to begin in 2016.

Schedule reserve for the sunshield was reduced to 9.25 months—two weeks from the critical path—due to various manufacturing challenges, and additional reserves will likely be needed in the near future. For example, an anomaly with the membrane retention devices—which need to operate correctly to ensure that the sunshield can unfold properly— during qualification testing required a redesign of the parts. According to contractor officials, when the devices were released, the contact between the metal surfaces moving adjacent to one another resulted in a small amount of debris being generated. Project officials expressed concern that the debris posed a risk of damaging other parts of the telescope. A new design has since been tested and proven to no longer pose the same risk. Additionally, coordinating the testing of the five individual layers of the sunshield created some delays. The five layers of the sunshield are currently in various stages of assembly, with two layers having been delivered to Northrop Grumman from the supplier in April 2015 and November 2015, respectively. In addition, in October 2015, the project reported that a piece of flight hardware for the sunshield’s mid boom assembly was irreparably damaged during vacuum sealing in preparation for shipping. The effect of the accident on the schedule has not yet been determined as project and prime contractor officials are currently determining the path forward.

Cryocooler Delivered but Remains a Significant Schedule Risk during Testing

The cryocooler continued to experience technical challenges in 2015 that used schedule reserve and delayed its delivery. Although it has now been delivered—approximately 18 months later than planned—the cryocooler remains a schedule risk as it begins testing. Northrop Grumman delivered the compressor assembly—the third and final cryocooler component to be delivered after the cold head assembly and electronics assembly—to JPL in July 2015. Over the last several years, the project has accommodated a series of cryocooler schedule slips by reordering and compressing JPL’s test schedule and resequencing the spacecraft bus integration schedule. For example, several tests that were initially planned to be conducted on flight hardware will now be conducted on the spare hardware later into JWST’s integration and testing phase and closer to launch. Additionally, in May 2015, the project used 3 weeks of reserve and removed 3 weeks of lower priority and redundant items from the planned 40 weeks of acceptance and end-to-end testing. The project took these actions to accommodate a further delay in the delivery of the compressor assembly. According to contractor officials, the delay was primarily caused by the contractor not scheduling enough time to complete the bake out—a process whereby moisture is removed by heating the compressor and pumping helium through it. Table 1 below shows the tests removed from the acceptance and end-to-end testing.

According to JPL officials, the thermal tests removed were from those that would have tested the cryocooler compressor assembly with the electronics assembly. A program official stated that the risk of eliminating these tests has been reduced now that the compressor and electronics assembly have been tested together for the first time. NASA and JPL assessed the removal of the electromagnetic interference and electromagnetic compatibility testing as low risk, and a program official stated that additional parallel activities and testing of the spare electronics to further mitigate the risk have been added. However, various integration and test experts we spoke with noted that eliminating testing is a sign that the project may be taking on additional risk because discovery of issues may be pushed to higher levels of testing or to later in the testing phase, where problems are more costly and time consuming to address.

To accommodate any delays to testing or problems that may be found, JPL currently maintains 12 weeks of reserve for acceptance and end-to- end testing of the cryocooler. According to JPL officials, a key driver in deciding to eliminate testing instead of using additional schedule reserve was to retain as much as possible in the event that a test has to be stopped and restarted—which would require approximately 5 weeks—in addition to the time it takes to mitigate a problem. At the completion of the acceptance and end-to-end testing programs, the cryocooler is needed for spacecraft integration and testing—when the spacecraft and sunshield are integrated—no later than August 2, 2016. Spacecraft integration and test is followed by the final observatory level integration and test— completing the telescope—which is expected to begin in September 2017. The cryocooler’s testing flow and schedule reserve leading to its integration with the spacecraft is depicted in figure 5 below.

Because the development and delivery of the cryocooler by Northrop Grumman took significantly longer than expected and to maintain the 12 weeks of reserve, JPL must complete acceptance and end-to-end testing in a more schedule-compressed environment. However, challenges have persisted in bringing the cryocooler flight model to testing and completing development of the spare model which could be needed if the flight model is not available in time to be integrated into the observatory for launch. For example, despite having an extra 18 months to prepare for the cryocooler testing due to the delay in the delivery of the compressor assembly, the project noted concerns with JPL’s readiness to accept the flight hardware. Specifically, a procedure error led to an interruption of the ongoing testing of the electronics assembly and fit checks of the flight cooler tower assembly and flight refrigerant line deployable assembly were delayed because the procedures were late in being completed.

Unusual Complexity of the JWST Project Heightens the Importance of Integration and Testing Phase to Reduce Substantial Risks

JWST is one of the most technologically complex projects NASA has undertaken. The project incorporates nine critical technologies— technologies that are required for the project to successfully meet requirements—whereas we found the average technology development project at NASA incorporates an average of 2.3 critical technologies. JWST also incorporates 15 pre-existing technologies that are being leveraged from previous development efforts.

Future testing on JWST has to reduce a significant amount of risk before the October 2018 launch. The project identifies and maintains a list of risks—currently with 102 items—that need to be tested and mitigated to an acceptable level in the next 3 years. According to the project, approximately 25 of these risks are not likely to be closed until the conclusion of the observatory integration and test phase—just prior to project launch. This is the point where the project has determined that no further mitigations are feasible and that these risks have been tested per a plan to reduce the risk, when possible, to an acceptable level. In some cases, it may take years to resolve a particular risk. For example, the project continues to track a risk related to the release mechanisms that hold the spacecraft and the OTE together for launch. Once in space, they are to activate and release to allow the OTE to separate from the spacecraft. If the mechanisms do not operate correctly, mission failure will occur. This risk was identified in January 2014 at the spacecraft critical design review. During testing, these devices were causing excessive shock when performing their releasing function. After redesign, the project is continuing to work on resolving the underlying problem and qualifying the new design. Additionally, while testing the redesigned mechanism, a new concern arose that it could release early which would cause mission failure. According to a program official, the redesigned mechanism is not needed for spacecraft integration and test until summer of 2016 and therefore these issues do not pose a significant schedule concern at this time.

As integration and testing moves forward, the project will need to be able to resolve problems in a timely manner to stay on schedule. A backlog of unresolved problems and risks may indicate there may not be enough schedule left before launch to complete all necessary work. The project keeps track of these problems via problem reports and problem failure reports. Thus, ensuring that problem reports and problem failure reports are resolved in a timely manner is key to successfully launching the project on time. Project officials reported that they do not track problem report and problem failure report closure rates over time; instead, they monitor the reports to ensure that they are closed before subsequent test events and receive monthly briefings from the contractors on the status of their progress. According to project officials responsible for the ISIM and OTIS development and testing, while there are numerous problem reports open at any given time, they are comfortable with the number of open reports at this stage of the project. Northrop Grumman officials reported that development of the OTE, sunshield, and spacecraft are on track with respect to problem and failure reports, which they refer to as nonconformance reports. Additionally, experts we spoke with told us that addressing requests for action (RFAs) from project reviews is important because RFAs are written to identify potential risks to the project. Since the spacecraft critical design review in January 2014, the project has closed 14 RFAs from that review while one related to the release mechanism noted above remains open. The project tracks and reports open RFAs to senior management at NASA, and we will continue to examine the open RFA and additional RFAs that result from future reviews to monitor their timely closure.

The extent of JWST’s deployments—which are necessary because JWST must be stowed for launch to fit in the launch vehicle—means the telescope could fail to operate as planned in an extensive number of ways. According to project officials, there are over 100 different ways that a failure could occur, referred to as single point failure modes, across hundreds of individual items in the observatory. Each of these could result in a loss of minimum mission objectives, and thus needs to be fully tested and understood. Nearly half of the single point failure modes involve the deployment of the sunshield. The approval of single point failures requires written justification from the project including sound engineering judgement, supporting risk analysis, and implementation of measures to mitigate the risk to acceptable levels. The project’s mission systems engineers have developed justifications and mitigation strategies for its single point failures, and project officials expect these to be summarized and submitted to the agency prior to launch. According to project officials, this approach is consistent with other high-priority NASA missions, which require the most stringent design and development approach that NASA takes to ensure the highest level of reliability and longevity on orbit.

Project Continues to Meet Cost Commitments but Contractor Workforce Increases and Unreliable Contractor Performance Data Pose Management Challenges

The JWST project continued to meet its cost commitments throughout fiscal year 2015 despite cryocooler delays that used a disproportionate amount of cost reserves. However, the project required larger than planned workforce levels to complete new and existing work, which poses a cost threat in future years if levels do not decrease. To help manage the project and account for new risks since the 2011 replan, JWST project officials conducted a cost risk analysis of the Northrop Grumman contract. We found that while the cost risk analysis substantially met best practices, these officials do not plan to periodically update it. Instead, the project is using risk-adjusted analyses to update and inform its cost position. However, we found that this method is a simplified version of a cost-risk analysis that does not contain the same rigor or allow the project to prioritize risks. Furthermore, we found anomalies in the contractor- provided data rendering the results of the analyses unreliable. Finally, we also found the project lacks an independent surveillance mechanism for the data to ensure anomalies are corrected by the contractor before being incorporated into larger analyses. As a result, the project is relying partially on unreliable information to inform its cost and schedule decision-making.

Project Continues to Stay on Budget and Spend within Cost Commitment Despite Significant Cryocooler Cost Growth

Project officials managed JWST within its allocated budget for the fourth consecutive year since the 2011 replan. Additionally, the project’s fiscal year 2016 budget request to Congress is consistent with its cost commitment. According to preliminary estimates, at the end of fiscal year 2015, the project spent $68 million dollars more than planned at the beginning of the fiscal year, carrying over less money into fiscal year 2016 than originally planned. As in past years, the project used a portion of its cost reserves to address technical challenges that included funding activities to address significant delays with the cryocooler. The project also used program-level cost reserves to pay for new work that included conducting additional thermal verification tests and risk reduction activities, such as an analysis to better understand how JWST will likely interact with its launch vehicle—the Ariane 5.

The cryocooler used a significant share of the project’s fiscal year 2015 cost reserves—more than 50 percent—to fund the workforce for this effort and address technical issues. This is the fourth year in a row that the cryocooler used a substantive portion of the project’s cost reserves to further fund the subcontractor’s schedule delays in delivering its components. The project estimates that the overall cryocooler development cost will be nearly 250 percent higher than baselined at the 2011 replan. The Northrop Grumman cryocooler team forecasts that a larger workforce is needed until at least February 2016 when the spare compressor assembly is currently scheduled to be delivered. JPL will maintain the majority of its workforce through the conclusion of spare cryocooler testing. After testing concludes, its workforce is projected to decrease by about 50 percent. Project cost reserves will likely continue to be needed to fund cryocooler development and testing costs until fiscal year 2017 when JPL testing of the spare compressor assembly is scheduled to conclude.

Larger Than Planned Workforce Poses a Threat to Meeting Cost Commitment

While the project remains on cost, contractor work is costing more to complete because a larger workforce than planned was needed for components beyond the cryocooler, including Northrop Grumman for the sunshield, spacecraft, and OTE, and Harris for OTIS testing and preparation. This need derives from work taking longer than planned to complete and additional work requested by NASA. For example, Northrop Grumman’s workforce projections for fiscal year 2015 predicted a peak in the workforce in November 2014. However, the actual workforce peaked in February 2015 and continued to remain above the projected peak until August 2015. While workforce numbers have declined somewhat since February, these increases largely remained in place through the end of the fiscal year. In its role as prime contractor, Northrop Grumman’s workforce stayed within its budget in fiscal year 2015. From January through July of 2015, its workforce was exactly at its funding threshold in order to conduct new work and address technical issues for its body of work. In addition, larger workforces contributed to additional contractor cost for two other development efforts—OTIS testing and the cryocooler—requiring the use of additional project cost reserves.

Looking forward, the primary threat to JWST meeting its long-term cost commitment is the prime contractor, which must continue to control its costs and decrease its workforce. For the past 20 months, Northrop Grumman’s actual workforce exceeded its projections. Figure 6 below illustrates the difference between the workforce levels that Northrop Grumman projected at the beginning of fiscal years 2014 and 2015 and its actual workforce levels for those periods.

Based on its projections at the beginning of the fiscal year, Northrop Grumman exceeded its total fiscal year 2014 workforce monthly projections by about 12 percent, and exceeded its projections for fiscal year 2015 by about 20 percent. On average, in fiscal year 2015, Northrop Grumman was 121 FTEs above its projections each month, and at the end of fiscal year 2015, it exceeded its monthly projection for September 2015 by 235 FTEs. While actuals have remained above projections since the workforce levels peaked in February 2015, Northrop Grumman currently projects that its workforce will decline throughout fiscal year 2016, with the exception of August 2016, when additional work is projected to be needed for integration and testing, among other areas. However, this was the projection for both fiscal years 2014 and 2015 and has yet to happen. For example, while Northrop Grumman expected to be ramping down by the end of fiscal year 2014, its projections at the start of fiscal year 2015 were approximately 55 percent higher than where workforce levels were projected for the end of fiscal year 2014. The primary drivers that have increased the cost and size of the workforce under the prime contract have been the development of the sunshield and spacecraft and additional work NASA has requested. Over 60 percent of the cost increases are attributed to addressing technical concerns such as sunshield alignment and verification work, mechanical design integration, and spacecraft mass reduction. Northrop Grumman has covered additional costs pertaining to technical issues through its management reserves, and has not needed project cost reserves in fiscal years 2014 and 2015. The remaining cost increases are attributable to new contract scope which has been funded by the JWST program. Some of this new scope included additional spacecraft simulators, as well as new thermal risk reduction testing to verify the final design changes made to the core of the telescope–the region between all the observatory elements.

Approximately 15 percent of work remains on Northrop Grumman’s contract and its management reserves exceed the recommended minimum amount that should be held at the contractor level—10 percent or more of the cost of work remaining on the project. Significant decreases in the workforce are planned to occur in fiscal year 2017 when final hardware delivery to observatory integration and test is scheduled to take place. To incentivize the contractor to lower its workforce, project officials evaluate workforce management as part of NASA’s appraisal of Northrop Grumman’s performance in its award fee determinations. The project also communicates frequently with the contractor including phone calls, face to face meetings twice a month, and quarterly in-person management meetings to discuss workforce planning, among other subjects. The project has communicated the need to reduce the workforce size, but since Northrop Grumman has operated within its budget in fiscal years 2014 and 2015, the award fee it has received has not been reduced as a result of workforce size issues.

The subcontractor for OTIS testing, Harris, needed additional funding to cover cost overruns and additional work. Project cost reserves were utilized to pay for this work to maintain schedule through a contract change in January 2015. Over 55 percent of the increase was made to address cost overruns that resulted from increasing workforce levels to maintain schedule. The rest of the contract increase covered new scope. As a result, Harris is anticipating more work than originally planned for fiscal years 2016 and 2017. Despite the contract increases, Harris’s management reserves are 2.5 percent as of August 2015—significantly below the 10 percent cost of work remaining that is considered to be healthy. With over 25 percent of work remaining on its contract, this low level of reserves means that any additional overruns will likely need to be covered by project-held reserves.

NASA’s JWST 2014 Cost Risk Analysis Substantially Met Best Practices

We found that NASA’s 2014 cost risk analysis on Northrop Grumman’s remaining work substantially met best practices. In December 2014, we recommended that project officials update the 2011 JWST cost risk analysis utilizing best practices, and to update it periodically as significant risks emerge. NASA partially concurred with our recommendation stating that the program and project use a range of tools to assess the performance of the project and conducted a one-time update to the cost risk analysis in 2014. We found that NASA’s updated cost risk analysis substantially met best practices. For example, it incorporated subject matter expert input to model cost and schedule uncertainties from the prime contractor’s threats and opportunities list—both of which are components of the best practice of modeling a probability distribution for each cost element’s uncertainty based on data availability, reliability and variability. See appendix I for a list of best practices that we used to evaluate cost risk and uncertainty. In addition, NASA included correlation between elements to account for different cost elements being affected by the same external factors—another best practice.

However, the cost risk analysis also had some weaknesses as a result of not fully following best practices. For the first best practice noted above on modeling probability distribution, NASA relied on the contractor’s risk data without conducting corroborating interviews with contractor personnel to obtain insight into threats and opportunities not listed in contractor data. For the same best practice, the detailed schedule that reflected all of the work that needed to be done by Northrop Grumman that was used for the cost risk analysis had some activity sequencing logic issues. For example, we found instances where activities listed were not sequentially linked to one another. As a result, this called into question the calculation of the critical path during simulations as well as the ability of the schedule to dynamically respond to changes, which it must do thousands of times during the risk simulations. Moreover, the JWST project does not plan to periodically update its cost risk analysis even as additional risks have emerged. JWST officials stated that the cost risk analysis was a time intensive process to complete and that the program and project use various tools consistent with best practices to assess all major contractors’ performance. Nonetheless, best practices call for conducting periodic updates to a cost risk analysis as a project progresses even if it is not experiencing problems. Updating the cost risk analysis is also part of the best practice of implementing a risk management plan with the contractor which calls for identifying and analyzing risk, planning for risk mitigation, and continually tracking risks. An accurate cost risk analysis is particularly vital to JWST because about 70 percent of the project cost reserves have been used to address concerns that were not anticipated as threats by the project’s budget system. Failure to update the cost risk analysis as we recommended in 2014 limits stakeholder confidence that the cost risk analysis prepared in 2014 accurately reflects the project’s current financial status. Given this uncertainty, it is important for the project to have reliable information for the risks that are known to inform decision making.

Unreliable EVM Data Limit Project’s Ability to Reliably Forecast Future Cost Growth for Contractors

One of the tools that the project has started to use in place of updating the cost risk analysis is a monthly risk-adjusted analysis to provide insight into potential future cost growth. The monthly risk-adjusted analyses are based on contractor EVM data that incorporate known threats to provide an estimate at completion (EAC) that is updated monthly by NASA for each contractor. The results of these analyses may then be compared to the contractors’ estimates and project cost reserves to provide insight into the project’s ability to cover future increases. Monthly risk-adjusted analyses demonstrate a commitment by NASA to manage and project future costs.

However, we found that the risk-adjusted analyses do not serve as an adequate substitute for an updated cost risk analysis because they are a simplified version of a cost risk analysis that does not allow the project to prioritize risks or assign confidence levels to meet key milestones in the schedule consistent with best practices for cost risk analyses. Additionally, based on our analysis of contractor EVM data over 17 months, we found that some of the data used to conduct the analyses were unreliable. First, we found that both Northrop Grumman and Harris were reporting optimistic EACs at the time of our analysis that did not align with their historical EVM performance and fell outside the low end of our independent EAC range. Second, we found various anomalies in contractor EVM data for both the Northrop Grumman and Harris work that they had not identified throughout the 17-month period we examined. The anomalies included unexplained entries for negative values of work performed (meaning that work was unaccomplished or taken away rather than accomplished during the reporting period), work tasks performed but not scheduled, or actual costs incurred with no work performed. For Northrop Grumman, many were relatively small in value ranging from a few thousand to tens of thousands of dollars. These anomalies are problematic because they distort the EVM data, which affect the projection of realistic EACs. We found that these anomalies occurred consistently within the data over a 17-month period, which brings into question the reliability of the risk-adjusted EAC analysis built upon this information. NASA did not provide explanations into the anomalies for either contractor. While the contractors were able to provide explanations for the anomalies upon request, their explanations or corrections were not always documented within EVM records. Some of the reasons the contractors cited that were not in the EVM records included tasks completed later than planned, schedule recovered on behind schedule tasks, and replanning of customer-driven tasks. Finally, like the cost-risk analysis in 2014, the risk-adjusted EAC analysis does not include interviews with contractor officials to gain insight into risks which may not be present in the contractors’ threats and opportunities list. Without updating the cost risk analysis, reconciling and documenting data anomalies, and utilizing reliable data for the risk-adjusted EAC, the JWST project does not have a reliable method to assess its cost reserve status going forward. This means that some of the cost information the project officials use to inform their decision making may indicate they are in good shape when the reality might be otherwise, and as result, project management may not have a solid basis for decision making.

In discussions with the contractors, we found that the project also lacks an independent surveillance mechanism, such as the Defense Contract Management Agency, to monitor contractors’ EVM data—provided to the project each month from two of the contractors. Surveillance entails reviewing a contractor’s EVM system with the purpose of focusing on how well a contractor is using its EVM system to manage cost, schedule, and technical performance. However, the lack of surveillance and the data anomalies in EVM data are problems we previously identified across NASA’s portfolio of major spaceflight projects. We found in November 2012 that 4 of NASA’s 10 major spaceflight projects we reviewed had established formal independent surveillance reviews. For the 6 projects that did not have formal independent surveillance in place, we found that each provided evidence that they instituted monthly EVM data reviews, which according to project officials, helped them to continually monitor cost and schedule performance. However, we found that the rigor of both the formal and informal surveillance reviews was questionable given the numerous EVM data anomalies we found in the monthly EVM data. As a result, we recommended that NASA improve the reliability of project EVM data by requiring projects to implement a formal surveillance program that ensured anomalies in contractor-delivered and in-house monthly earned value management reports were identified and explained, and report periodically to the center and mission directorate’s leadership on relevant trends in the number of unexplained anomalies. Citing resource constraints, NASA partially concurred with the recommendation and commented that it did not plan to implement a formal surveillance program, but agreed that the reliability and utility of the EVM data needed to be improved and noted several steps it planned to take to do so. We continue to believe that implementing this recommendation would be beneficial and prevent anomalies in EVM data from occurring that we have identified on the JWST project. Implementing surveillance of EVM contractor data is a best practice listed in the NASA Earned Value Management Implementation Handbook and GAO’s Cost Estimating and Assessment Guide. With adequate surveillance in place, the anomalies we found in the EVM data could have been identified earlier and corrective action could have been directed to the contractors to explain the anomalies in the data. Without implementing proper surveillance, the project may be utilizing unreliable EVM data in its analyses to inform its cost and schedule decision making.

Independent Oversight Has Been Built into JWST’s Development Life Cycle to Improve the Likelihood of Mission Success

NASA has taken steps to provide independent oversight of the JWST project. Independent oversight of the JWST project has played and will likely continue to play an important role leading up to JWST’s launch in October 2018. Before the 2011 replan, two groups examined JWST to address underlying concerns with schedule and cost and made recommendations that NASA implemented. On an ongoing basis until launch, the Standing Review Board and the Independent Verification & Validation (IV&V) facility are to continue to oversee progress on hardware and software development, identify concerns, and assist the project to identify solutions to reduce risk and improve JWST’s likelihood of success.

Independent Oversight Has Provided Insight to Reduce Risk

Various groups internal and external to NASA have conducted reviews, provided insights, and identified schedule efficiencies to inform and enhance the project’s approach to managing the development of JWST. Prior to the 2011 replan and because of concerns raised at the JWST mission critical design review held in the spring of 2010, the Test Assessment Team was formed to address those concerns. Convened by the Astrophysics division of the NASA Science Mission Directorate, the team included nine members and three NASA consultants with considerable experience in systems engineering, instrument development, system verification, modeling and testing, and other areas focused on reviewing plans for the ISIM and OTIS cryogenic testing. The team was primarily tasked to determine whether (1) the test plans in place at that time were sufficient to test the relevant observatory functions, (2) the key optical and thermal objectives were clearly identified, (3) the test plans themselves were properly scoped and prioritized, (4) any duplicative or unnecessary tests existed in the plans, and (5) the plans were overly ambitious or optimistic regarding hardware performance and analysis capabilities. Their insights and recommendations have helped to decrease programmatic cost and future growth as well as to find schedule efficiencies. For example, they recommended OTIS testing duration be reduced from 167 to 90 days while still verifying critical functions of the telescope. Also prior to the 2011 replan, the Chair of the Senate Subcommittee on Commerce, Justice, Science, and Related Agencies asked that NASA set up a panel to review the JWST project because of concerns about cost growth and schedule delays. In response, NASA convened the Independent Comprehensive Review Panel to provide an independent, integrated perspective and response with the goal of providing recommendations that would lead to a successful launch while minimizing cost. At the conclusion of its work in October 2010, the panel made 22 recommendations to NASA to increase oversight, improve communications, and assist with risk management and mitigation, among other recommendations. NASA implemented all of these recommendations. Both of these reports have informed our ongoing reviews of the JWST project as we have incorporated many of the concerns on cost estimates and cost reserves into our methodology and reporting on the health and status of JWST as it moves forward.

Independent Standing Review Board Provides Additional Expertise and Insight to Help Manage Project

Another aspect of independent oversight that is a key element of NASA’s strategic framework for managing space flight projects are Standing Review Boards which consist of technical experts who do not actively work on a specific project or program. The mission of the boards is to provide NASA senior management with objective information to ensure there is appropriate program and project management oversight to increase the likelihood of mission success. The boards help to determine the adequacy of programs’ (1) management approach, (2) technical approach, (3) integrated cost and schedule estimates and funding strategy, and (4) risk management, among others. NASA’s Independent Program Assessment Office and various NASA centers organize these boards and coordinate their involvement at different reviews. The boards are involved at various agency-level reviews with some members participating in lower-level reviews at NASA’s different centers, in monthly reviews held by the projects and program, or in special reviews on a specific topic or set of issues. Standing Review Boards may also make non-binding recommendations after life-cycle reviews, but do not have programmatic or technical authority over the programs or projects. The Standing Review Board Handbook describes three types of boards that may be formed to provide independent oversight of programs or projects. See table 2 below for the three types of Standing Review Boards.

NASA’s Standing Review Board Handbook states that a civil servant consensus with no expert support is the preferred structure within NASA because experience demonstrates that a consensus board leads to a more meaningful discussion of the review findings and recommendations, especially where dissenting opinions are discussed. A non-consensus mixed board provides only the perspective of the chairman. In 2015, 1 of 33 active Standing Review Boards was a civil service consensus board with no expert support, 15 were civil service consensus with consultant support, and 17 were non-consensus mixed boards. Although NASA guidance prefers civil servant consensus boards, NASA officials told us that they have found it challenging to staff boards exclusively with civil servants for a number of reasons including availability of staff, finding a person with the appropriate skill set, and independence reasons, among others.

JWST has had a number of changes occur on the boards overseeing the project for different reasons. JWST has had a Standing Review Board since 2006 when a special review was conducted. During that review and from 2008 to 2014, the board was a non-consensus board led by an outside expert chosen by NASA senior officials. The experts were civil servants as well as non-civil-servant experts. In May 2014, the chairman retired, a new chair was appointed the same year, and NASA senior officials changed the board to a consensus board with consultants. Independent Program Assessment Office officials told us that board types can change for numerous reasons, including when a project or program enters a different phase of development that may require different technical skills or if all of the convening authorities request it. As a result of the retirement of the chairman, most of the 2008-2014 Standing Review Board members who were not civil servants but who had overseen JWST for more than 6 years were replaced and 2 civil servants were carried over to the new board. Consultant support was added for schedule analysis and in one technical area to support launch vehicle integration because NASA has never launched a mission on an Ariane 5 rocket as it plans to do for JWST. Before retiring in 2015, the previous Standing Review Board chairman expressed the importance of having representation from JPL as a member of the board to provide experience working on unmanned spacecraft projects—but a JPL member could not be added since JPL employees are not civil servants and can only be consultants to the board. With the appointment of a new chairman in October 2015, there have been additional membership changes to the board including the addition of a JPL consultant.

Independent Oversight of Critical Software Confirms Development Is on Schedule with Concerns Regarding Large Number of Developers

NASA’s IV&V facility—which independently examines software development—reviews mission critical software for most NASA programs and projects to achieve the highest levels of safety and cost-effectiveness by ensuring that developed software will perform as required. Experts at the facility work to uncover high-risk errors early in the development life cycle of software for many NASA programs and projects. IV&V is a process whereby organizations can reduce the risks inherent in system development and acquisition efforts by having a knowledgeable party who is independent of the developer to determine whether the system or product meets the users’ needs and fulfills its intended purpose. IV&V applies software engineering best practices to risk elements on safety critical and mission-critical software throughout the development life cycle. We have found IV&V to be a leading practice for federal agencies in managing their complex, large-scale, or high-risk acquisition of programs. Software development is a challenge we have found on many different acquisitions—some space-related—in government programs that has led to schedule delays and cost growth. Examples include the F-35 Joint Strike Fighter, the Aegis Modernized Weapon System, NASA’s Stratospheric Observatory for Infrared Astronomy, and Geostationary Weather Satellite development, among others.

The goal of IV&V is to examine the three following questions regarding software:

Will the system do what it is supposed to do?

Will the system not do what it is not supposed to do?

Will the system perform as expected under adverse conditions?

IV&V is required to examine software on all projects with a life cycle cost over $1 billion, other projects over $250 million with a higher risk classification, or those specifically selected by the NASA Chief, Safety and Mission Assurance. Once selected, a portfolio-based risk assessment is developed to identify top-level mission capabilities and a risk based assessment process identifies the most important system capabilities and the software components that play the most important role in the mission. IV&V officials noted that due to limited resources, they examine mission and safety critical software and they do not have the funding to examine all programs or projects across NASA’s portfolio.

Generally, IV&V officials stated that they believe JWST’s software development is going well, but the testing that lies ahead—when the different components are integrated—will be a challenge. For example, IV&V officials noted that JWST’s software build is the largest they have reviewed for a science mission, but not the largest they have reviewed across NASA as some Human Exploration Operations are larger. They said that most of JWST’s software required to position and deploy the telescope will be examined by IV&V. However, they noted that JWST’s integration is more challenging, primarily due to the number of software developers involved. While most science programs or projects have two to four software developers, JWST has eight. JWST’s software development has been examined by IV&V since fiscal year 2004 and, according to officials, will likely continue to be examined until after launch when operations begin. IV&V officials said they do not examine the software for the launch vehicle.

While IV&V’s function requires independence from programs and projects, there have been recent changes in funding that have reduced its financial independence to some extent. Organizationally, the IV&V Facility remains independent by reporting to the Office of the Director of Goddard and the Office of Safety and Mission Assurance at NASA Headquarters— not to the programs or projects it examines. However, financially, starting in 2015, an IV&V financial management official said that 75 percent of the IV&V’s funding came from NASA Headquarters via the Agency Management Operations fund and the remaining 25 percent was divided amongst the various mission directorates. This changed from the past 10 years, when 100 percent of the IV&V Facility’s budget came from the Agency Management Operations to ensure the independence of the IV&V office. We have previously found that financial independence requires that the funding for IV&V be controlled by an organization separate from the development organization. This ensures that the effort will not be curtailed by having its funding diverted to other program needs, and that financial pressures cannot be used to influence the effort. As a user of IV&V’s expertise, JWST, via the use of program cost reserves, contributed a small portion of funding to the software IV&V facility to help fund their budget in fiscal year 2015. While this financial situation was new in fiscal year 2015, we will continue to monitor how NASA deals with funding the IV&V facility in the future to protect its independence.

Conclusions

The JWST project has made progress building, integrating, and testing significant portions of JWST in the past year within the commitments made at the time of the 2011 replan for both schedule and cost. With the third major integration and test period starting in 2016, resolving technical challenges in a timely manner, and ensuring the OTIS test goes smoothly are key to continued progress within the project’s schedule commitment. Additionally, reducing the size of Northrop Grumman’s workforce and controlling costs within the fiscal year 2016 budget will be key metrics to monitor to demonstrate the project can meet requirements within its cost commitment. In the past, the project has benefited from independent expertise, information, and recommendations to improve the management of the project.

Moving forward, the project may benefit from having more reliable data provided from its contractors to ensure that its EACs, which take into account risks and threats, are better able to inform its cost status. While the contractors were able to explain the anomalies, most had not been previously identified or documented. NASA used the data for its analyses, which subsequently raised questions about the reliability of those analyses. Making management decisions using unreliable data can result in bad decision making and can misinform the project on its long-term financial position which may have significant consequences if not corrected. We recommended in our December 2014 report that NASA conduct a cost risk analysis and follow best practices, which include updating it as risks change during the life of the program. Because the project is not going to conduct another cost risk analysis, putting independent surveillance in place to improve the accuracy of its risk- adjusted analysis—despite its weaknesses relative to the information a cost risk analysis provides—will provide better information to inform its decision making. In November 2012, we recommended that NASA improve the reliability of project EVM data by requiring its major spaceflight projects to implement a formal surveillance program that ensured anomalies in contractor-delivered data and in-house monthly EVM reports were identified and explained. NASA partially concurred with this recommendation but has not taken steps to require surveillance on projects like JWST. However, we continue to believe that improving the surveillance on projects will help reduce data anomalies from occurring like the ones we identified on JWST, resulting in better information and analyses to inform project decision making.

Recommendation for Executive Action

To resolve contractor data reliability issues and ensure that the project obtains reliable data to inform its analyses and overall cost position, we recommend that the NASA Administrator direct JWST project officials to require the contractors to identify, explain, and document all anomalies in contractor-delivered monthly earned value management reports.

Agency Comments and Our Evaluation

We provided a draft of this report to NASA for comment. In written comments, NASA agreed with our recommendation. These comments are reprinted in appendix IV. NASA also provided technical comments, which have been addressed in the report, as appropriate.

We are sending copies of the report to NASA’s Administrator and interested congressional committees. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

Should you or your staff have any questions on matters discussed in this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to assess (1) the extent to which technical challenges have impacted the James Webb Space Telescope (JWST) project’s ability to meet its schedule commitments, (2) the current cost status of the JWST project and the primary challenges that may influence the project’s ability to meet its future cost commitments, and (3) the extent to which independent oversight provides insight about project risks to management.

To assess the extent to which technical challenges have impacted the JWST project’s ability to meet its schedule commitments, we reviewed project and contractor schedule documentation, and held interviews with program, project, and contractor officials on the progress made and challenges faced building the different components of the telescope. We examined and analyzed monthly JWST project status reports to management to monitor schedule reserve levels and usage and potential risks and technical challenges that may impact the project’s schedule, and to gain insights on the project’s progress since our last report in December 2014. Further, we attended flight program reviews at the National Aeronautics and Space Administration (NASA) headquarters on a quarterly basis where the current status of the program was briefed to NASA headquarters officials outside of the project. We examined selected individual risks for elements and major subsystems from monthly risk registers prepared by the project to understand the likelihood of occurrence and impacts to the schedule based on steps the project is taking to mitigate the risks. We examined previous and current test schedules and plans to understand the sequence, what risks will be mitigated, which risks will continue, and any reductions to planned testing. Furthermore, we interviewed experts within and outside of NASA to identify criteria, best practices, and metrics that may be used to assess the project’s progress in reducing risk or provide insight into the health of the project. Finally, we interviewed project officials at Goddard, contractor officials from the Harris Corporation, the Jet Propulsion Laboratory, the Space Telescope Science Institute, and different divisions of Northrop Grumman Aerospace Systems concerning technological challenges that have had an impact on schedule, and the project’s and contractor’s plans to address these challenges.

To assess the current cost status of the JWST project and the primary challenges that may influence the project’s ability to meet its future cost commitments, we reviewed and analyzed program, project, contractor, and subcontractor data and documentation and held interviews with officials from these organizations. We reviewed JWST project status reports on cost issues to determine the risks that could impact cost. We analyzed contractor and subcontractor’s workforce plans against workforce actuals to determine whether contractors’ are meeting their workforce plans. We monitored and analyzed the status of program, and project cost reserves in current and future fiscal years to determine the project’s financial posture. We evaluated the cost risk analysis conducted by NASA of the remaining Northrop Grumman work to determine the extent to which all applicable best practices from GAO’s Cost Estimating and Assessment Guide were used to build the analysis. Those best practices included the following:

A probability distribution modeled each cost element’s uncertainty based on data availability, reliability, and variability.

The correlation between cost elements was accounted for to capture risk.

A Monte Carlo simulation model was used to develop a distribution of total possible costs and an S curve showing alternative cost estimate probabilities.

The probability associated with the point estimate was identified.

Contingency reserves were recommended for achieving the desired confidence level.

The risk-adjusted cost estimate was allocated, phased, and converted to then year dollars for budgeting, and high-risk elements were identified to mitigate risks.

A risk management plan was implemented jointly with the contractor to identify and analyze risk, plan for risk mitigation, and continually track risks.

We examined and analyzed earned value management (EVM) data from two of the project’s contractors to identify trends in performance, whether tasks were completed as planned, and likely estimates at completion. We also conducted analysis to ensure the reliability of the data over a 17- month period. In addition, we examined and analyzed risk-adjusted analyses from NASA to determine what information they provide to the project, the risks incorporated, their reliability, and how the project is utilizing this information. We also discussed our assessment of the project’s data and analysis with program and project officials to obtain their input.

To assess the extent to which independent oversight provides insight about project risks to management, we reviewed documentation and data from NASA relevant groups, the program, the project, and the Standing Review Board and held interviews with experts as well as officials from independent oversight entities. We analyzed NASA policy and guidance documents to understand the elements for setting up and managing a Standing Review Board. We also reviewed the Test Assessment Team and Independent Comprehensive Review Panel team reports to determine how independent oversight has provided insight to JWST in the past. We interviewed officials at NASA’s Independent Program Assessment Office, as well as past and current Standing Review Board members, to understand how Standing Review Boards are created, members are selected, and how structural and personnel changes are made over the life of NASA programs and projects, including JWST. We also interviewed and reviewed documentation and analysis provided by NASA’s Independent Verification and Validation group working on JWST’s software development to determine the extent to which this group is providing oversight of JWST software development, to determine the health of software development on JWST, and determine what kinds of problems remain. We did not independently review JWST’s software development.

Our work was performed primarily at NASA headquarters in Washington, D.C.; Goddard Space Flight Center in Greenbelt, Maryland; the Independent Verification and Validation facility in Fairmont, West Virginia; and by video teleconference with officials from the Independent Program Assessment Office at Langley Research Center, Hampton, Virginia. We also visited the Jet Propulsion Laboratory in Pasadena, California; Northrop Grumman Aerospace Systems in Redondo Beach, California; and the Space Telescope Science Institute in Baltimore, Maryland.

We conducted this performance audit from February 2015 to December 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Elements and Major Subsystems of the James Webb Space Telescope (JWST) Observatory

Appendix III: Organizational Chart for the James Webb Space Telescope (JWST) Program

Appendix III: Organizational Chart for the James Webb Space Telescope (JWST) Program The Jet Propulsion Laboratory is the contractor for the development of the cryocooler, but has subcontracted most of the work to a different division of Northrop Grumman than the one that is responsible for OTE, spacecraft, and sunshield development.

Appendix IV: Comments from the National Aeronautics and Space Administration

Appendix V: GAO Contact and Staff Acknowledgements

GAO Contact

Cristina Chaplain, (202) 512-4841 or chaplainc@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Shelby S. Oakley, Acting Director; Arthur Gallegos, Assistant Director; Jay Tallon; Assistant Director; Karen Richey, Assistant Director; Jason Lee, Assistant Director; Brian Bothwell; Patrick Breiding; Aaron Gluck; Laura Greifner; Michael Kaeser; Katherine Lenane; Silvia Porres; Carrie Rogers; Sylvia Schatz; and Ozzy Trevino made key contributions to this report.