James Webb Space Telescope: Project on Track but May Benefit from Improved Contractor Data to Better Understand Costs
Published: 2015-12-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: JWST is one of NASA's most complex and expensive projects, at an anticipated cost of $8.8 billion. With significant integration and testing scheduled in the 3 remaining years until the planned launch date, the JWST project will need to continue to address many challenges and identify problems, many likely to be revealed during its rigorous testing to come. The continued success of JWST hinges on NASA's ability to anticipate, identify, and respond to these challenges in a timely and cost-effective manner to meet its commitments.
Conference Report 112-284 included a provision for GAO to assess the project annually and report on its progress. This is the fourth such report. This report assesses (1) the extent to which JWST is meeting its schedule commitments and (2) the current cost status of the project, among other issues. To conduct this work, GAO reviewed monthly JWST reports, reviewed relevant policies, conducted independent analysis of NASA and contractor data, and interviewed NASA and contractor officials.
section_title: What GAO Found
paragraphs: The National Aeronautics and Space Administration's (NASA) James Webb Space Telescope (JWST) project is meeting its schedule commitments, but it will soon face some of its most challenging integration and testing. JWST currently has almost 9 months of schedule reserve—down more than 2 months since GAO's last report in December 2014—but still above its schedule plan and the Goddard Space Flight Center requirement. However, as GAO also found in December 2014, all JWST elements and major subsystems continue to remain within weeks of becoming the critical path—the schedule with the least amount of schedule reserve—for the overall project. Given their proximity to the critical path, the use of additional reserve on any element or major subsystem may reduce the overall project schedule reserve.
Before the planned launch in October 2018, the project must complete five major integration and test events, three of which have not yet begun. Integration and testing is when problems are often identified and schedules tend to slip. At the same time, the project must also address over 100 technical risks and ensure that potential areas for mission failure are fully tested and understood.
JWST continues to meet its cost commitments, but unreliable contractor performance data may pose a risk to project management. To help manage the project and account for new risks, project officials conducted a cost risk analysis of the prime contract. A cost risk analysis uses information about cost drivers, technical issues, and schedule to determine the reliability of a program's cost estimates. GAO found that while NASA's cost risk analysis substantially met best practices for cost estimating, officials do not plan to periodically update it. Instead, the project is using a risk-adjusted analysis to update and inform its cost position, but this analysis is a simplified version of a cost risk analysis—and not a replacement—and is based on contractor-provided performance data that contains anomalies that render the data unreliable. Further, the project does not have an independent surveillance mechanism, such as the Defense Contract Management Agency, to help ensure data anomalies are corrected by the contractor before being incorporated into larger cost analyses, as GAO recommended in 2012. As a result, the project is relying partially on unreliable information to inform its decision making and overall cost status.
section_title: What GAO Recommends
paragraphs: GAO recommends that the JWST project require contractors to identify, explain, and document anomalies in contractor-delivered monthly earned value management reports. GAO continues to believe that its 2012 recommendation to implement formal surveillance to help improve the reliability of contractor-provided data has merit and should be implemented. NASA concurred with the recommendation made in this report.

section_title: Letter
paragraphs: The James Webb Space Telescope (JWST) is one of the National  Aeronautics and Space Administration’s (NASA) most complex projects  and top priorities. The innovative technologies within the telescope as  well as the sheer size of some of its components—such as the tennis- court-sized sunshield—illustrate some of the immense challenges in  building it. While JWST has been an idea discussed and planned for  more than two decades, major pieces of the telescope are being built,  tested, and prepared for integration. The project has now entered into a  significant integration and test period that is expected to last until the  telescope is launched in the fall of 2018. It is in this period when elements  and major subsystems are integrated and tested in which unforeseen  challenges could arise and affect the cost and schedule for the project.  Until launch, NASA and its contractors’ ability to identify and respond to  future challenges in a timely and cost-effective manner will likely influence  whether JWST can meet its cost and schedule commitments to  Congress.
The on-time and on-budget delivery of JWST is also a high congressional  priority, as Conference Report 112-284 included a provision for GAO to  assess the JWST program annually and to report to the Committees on  Appropriations on key issues relating to program and risk management,  achievement of cost and schedule goals, program technical status, and  oversight mechanisms. This report is our fourth in response to that  provision. For this report, we assessed: (1) the extent to which technical  challenges have impacted the JWST project’s ability to meet its schedule  commitments; (2) the current cost status of the JWST project and the  primary challenges that may influence the project’s ability to meet future  cost commitments; and (3) the extent to which independent oversight  provides insight about project risks to management.
Our approach included an examination of the schedule, technical, and  cost performance of the project since our last report in December 2014— which focused on the project’s cost and schedule commitments, the  project’s lack of updated cost risk analyses for its major contractors, and  the project’s use of award fees to manage contractors—as well as the  extent to which independent oversight mechanisms are in place. To  assess the extent to which technical challenges have impacted the JWST  project’s ability to meet its schedule commitments, we reviewed project  and contractor schedule documentation, monthly status reports, selected  individual risks from monthly risk registers, previous and current test  schedules, and other documentation; and held interviews with program,  project, and contractor officials on the progress made and challenges  faced building the different components of the telescope. We also  interviewed experts within and outside of NASA to identify criteria, best  practices, and metrics that could be used to assess the project’s progress  in reducing risk or provide insight into the health of the project.
To assess the current cost status of the JWST project and the primary  challenges that influence the project’s ability to meet its future cost  commitments, we analyzed program, project, contractor, and  subcontractor cost data and documentation. We compared projected  workforce levels to actual workforce levels to determine differences and  their effect on cost. We also conducted an analysis on the earned value  management (EVM) data to ensure the reliability of the data over a 17- month period for two of the project’s contractors as discussed later in the  report. In addition, we assessed NASA’s 2014 cost risk analysis of  Northrop Grumman’s remaining work to determine whether it followed  best practices—such as whether modeling probability distributions were  based on data availability, reliability, and variability; simulations were run  to obtain a distribution of possible cost outcomes; and risk management  plans were implemented, among others—and how these and other tools  informed the project’s cost status. Appendix I identifies all of the  applicable best practices we used to assess the cost risk analysis.
To assess the extent to which independent oversight provided insight to  management about project risks, we reviewed NASA policies and  guidance documents to understand the elements for setting up and  managing a Standing Review Board; interviewed Independent Program  Assessment Office officials and past and current Standing Review Board  members; collected information and analysis from NASA’s Independent  Verification and Validation (IV&V) group; and interviewed IV&V officials.  See appendix I for a detailed description of our objectives, scope, and  methodology.
We conducted this performance audit from February 2015 to December  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: JWST is envisioned to be a large deployable, infrared-optimized space  telescope and the scientific successor to the aging Hubble Space  Telescope. JWST is being designed for a 5-year mission to find the first  stars and trace the evolution of galaxies from their beginning to their  current formation, and is intended to operate in an orbit approximately 1.5  million kilometers—or 1 million miles—from the Earth. With a 6.5-meter  primary mirror, JWST is expected to operate at about 100 times the  sensitivity of the Hubble Space Telescope. JWST’s science instruments  are to observe very faint infrared sources and as such are required to  operate at extremely cold temperatures. To help keep these instruments  cold, a multi-layered tennis-court-sized sunshield is being developed to  protect the mirrors and instruments from the sun’s heat. The sunshield  and primary mirror are designed to fold and stow for launch and fit within  the launch vehicle. When complete, the observatory segment of JWST is  to include several elements (Optical Telescope Element (OTE),  Integrated Science Instrument Module (ISIM), and spacecraft) and major  subsystems (sunshield and cryocooler). The JWST project is divided into  three major segments: the observatory segment, the ground segment,  and the launch segment. The hardware configuration created when the  Optical Telescope Element and the Integrated Science Instrument  Module are integrated, referred to as OTIS, is not considered an element  by NASA, but we categorize it as such for ease of discussion.  Additionally, JWST is dependent on software to deploy and control  various components of the telescope as well as collect and transmit data  back to Earth. The elements, major subsystems, and software are being  developed through a mixture of NASA, contractor, and international  partner efforts. See figure 1 below for an interactive graphic that depicts  the elements and major subsystems of JWST. For more information on  JWST’s organizational structure, see appendix III.
Given JWST’s complexity, integration and test activities are comprised of  five separate periods—two of which have already started—over the  course of almost 7 years to build the observatory. During the test periods,  the project works to mitigate risks to an acceptable level prior to launch.  According to project officials, while some risks may be eliminated entirely  through various mitigation strategies, others will be accepted as residual  risks that remain upon launch. See figure 2 below for the overall planned  integration and test flow for JWST that includes the remaining schedule  reserve—or extra time built into the schedule to address any issues  found.
For the majority of the work remaining, the JWST project will rely on three  contractors: Northrop Grumman, Harris (formerly Exelis), and the Space  Telescope Science Institute (STScI). Northrop Grumman plays the largest  role, developing the sunshield, the OTE, the spacecraft, and a cooling  subsystem for the Mid-Infrared Instrument (MIRI). Northrop Grumman  performs most of this work under a prime contract with NASA, but its work  on the MIRI cooler is performed under a separate subcontract with the Jet  Propulsion Laboratory (JPL). Harris is manufacturing the test equipment,  equipping the test chamber, and assisting in the testing of the optics of  JWST. Finally, STScI will collect and evaluate research proposals from  the scientific community and will receive and store the scientific data  collected, both of which are services that they currently provide for the  Hubble Space Telescope. Additionally, STScI is responsible for  developing the ground system that manages and controls the telescope’s  observations on behalf of NASA.
The MIRI instrument, one of the four instruments within ISIM, requires a  dedicated, interdependent two-stage cooler subsystem designed to cool  the infrared light detector to about 6.7 Kelvin (K), just above absolute  zero. This cooler is referred to as a cryocooler and works by moving  helium gas through about 10 meters (approximately 33 feet) of refrigerant  lines located on the sun-facing surface of the JWST observatory to the  colder, shaded side where the ISIM is located. According to NASA  officials, a cryocooler of this configuration has never been developed or  flown in space before. See figure 3 below for an illustration of the MIRI  cryocooler on JWST and the varying temperatures needed in different  areas of the telescope.
subsections: section_title: Cost and Schedule Reserves for NASA Projects
paragraphs: Complex development efforts like JWST face myriad risks and  unforeseen technical challenges which oftentimes can become apparent  during integration and testing. To accommodate these risks and  unknowns, projects reserve extra time in their schedules—which is  referred to as schedule reserve—and extra money in their budgets— which is referred to as cost reserve. Schedule reserve is allocated to  specific activities, elements, and major subsystems in the event there are  delays or to address unforeseen risks. Each JWST element and major  subsystem has been allocated schedule reserve. When an element or  major subsystem exhausts schedule reserve, it may begin to affect  schedule reserve on other elements or major subsystems whose  progress is dependent on prior work being finished for its activities to  proceed. The element or major subsystem with the least amount of  schedule reserve determines the critical path for the project. Any delay to  an activity that is on the critical path will reduce schedule reserve for the  whole project, and could ultimately impact the overall project schedule.
Cost reserves are additional funds within the project manager’s budget  that can be used to address unanticipated issues for any element or  major subsystem and are used to mitigate issues during the development  of a project. For example, cost reserves can be used to buy additional  materials to replace a component or, if a project needs to preserve  schedule reserve, reserves can be used to accelerate work by adding  shifts to expedite manufacturing and save time. NASA’s Goddard Space  Flight Center (Goddard)—the NASA center with responsibility for  managing JWST—has issued procedural requirements that establish the  levels of both cost and schedule reserves that projects must hold at  project confirmation. After this point, a specified amount of schedule  reserve continues to be required throughout the remainder of  development. In addition to cost reserves held by the project manager,  management reserves are funds held by the contractors that allow them  to address cost increases throughout development. We have found that  management reserves should contain 10 percent or more on the cost to  complete a project and are used to address different issues.
section_title: History of Cost Growth, Low Project Reserves, and Schedule Delays
paragraphs: JWST has experienced significant increases to project costs and  schedule delays. Prior to being approved for development, cost estimates  of the project ranged from $1 billion to $3.5 billion with expected launch  dates ranging from 2007 to 2011. Before 2011, early technical and  management challenges, contractor performance issues, low level cost  reserves, and poorly phased funding levels caused JWST to delay work  after confirmation, which contributed to significant cost and schedule  overruns, including launch delays. The Chair of the Senate Subcommittee  on Commerce, Justice, Science, and Related Agencies requested from  NASA an independent review of JWST in June 2010. In response, NASA  commissioned the Independent Comprehensive Review Panel, which  issued its report in October 2010, and concluded that JWST was  executing well from a technical standpoint, but that the baseline funding  did not reflect the most probable cost with adequate reserves in each  year of project execution, resulting in an unexecutable project. Following  this review, the JWST program underwent a replan in September 2011,  and Congress in November 2011 placed an $8 billion cap on the  formulation and development costs for the project. On the basis of the  replan, NASA rebaselined JWST with a life-cycle cost estimate of $8.835  billion that included additional money for operations and a planned launch  in October 2018. The revised life-cycle cost estimate included a total of  13 months of funded schedule reserve. In the President’s fiscal year 2013  budget request, NASA reported a 66 percent joint cost and schedule  confidence level—lower than the 70 percent level noted in NASA  procedural requirements—for these cost and schedule baselines. A joint  cost and schedule confidence level is the process NASA uses to assign a  percentage to the probable success of meeting cost and schedule  estimates and is part of the project’s estimating process.
section_title: Previous GAO Reviews of JWST Project
paragraphs: In December 2014, we found that the project was progressing within the  2011 replan for both cost and schedule. We reported on technical  challenges with JWST elements and major subsystems that had  consumed a portion of the cost and schedule reserves. We also found  that the cryocooler remained an ongoing challenge and continued to use  a disproportionate amount of cost reserves. Finally, we found that NASA  had not conducted a cost risk analysis since the 2011 replan. A cost risk  analysis determines the reliability of a program’s cost estimate by  determining a program’s cost drivers and the risk of cost overruns through  an analysis that links historical schedule information along with technical  issues and uncertainties in schedule and cost. Since new risks had  emerged, we recommended that NASA follow best practices when it  updated the 2011 analysis for the Northrop Grumman contract and  ensure the analysis is updated as significant risks emerge in the future.  NASA partially concurred with our recommendation and stated that the  JWST program and project use a range of tools to assess all major  contractors’ performance and that the project initiated a cost risk analysis  of Northrop Grumman’s contract incorporating best practices and would  update it when required by NASA policy. The status of NASA’s analysis  and our evaluation of it are discussed later in the report.
section_title: JWST Is Meeting Schedule Commitment with Majority of Testing and Challenging Integration Work Still to Come
paragraphs: The JWST project is currently on schedule with 8.75 months of schedule  reserve remaining. However, all of JWST’s elements and major  subsystems are within weeks of moving onto the project’s critical path,  potentially reducing schedule reserve further. This is a tenuous position  for the project given that it must complete five integration and test  periods, three of which have not yet started. Testing can uncover  problems that can be difficult or time-consuming to resolve, thereby  adding schedule risk to the project and the unusual complexity of JWST  further heightens these risks. To achieve mission success, the project will  have to address over 100 technical risks and ensure that the project’s  potential areas for mission failure are fully tested and understood before  project launch in October 2018.
subsections: section_title: Project Is Above Schedule Reserve Standards with Significant Testing on the Horizon
paragraphs: Overall project schedule reserve, currently at 8.75 months, remains  above Goddard requirements and the project’s plan—which was set  above the Goddard standard at the replan in 2011 and included more  reserve than required. However, as shown in figure 4 below, the use of  schedule reserve on any element or major subsystem—two of which have  entered integration and testing phases—may reduce the overall project  schedule reserve. While some use of schedule reserve is expected, the  proximity of each element and major subsystem schedule to the critical  path means that the project must prioritize the mitigations when problems  occur.
Overall, the project has used more than 30 percent of its schedule  reserve established at the time of the replan in 2011 to address technical  challenges. Our prior work has shown that it is in integration and testing  where problems are most likely to be found and as a result, schedules  tend to slip. As we found in 2012, the project has a set amount of time  allocated to the final three integration and test efforts over the next 3  years, with between 2 and 4 months for each. This time could easily be  used if a significant problem occurred. For example, the OTIS  integration and test period—the first major integration involving OTE and  ISIM—planned to start in 2016 currently has 3 months of schedule  reserve allocated at the end of testing. The final event in the OTIS  integration and test effort is a cryovacuum test that takes approximately 3  months to complete. If an issue occurred that required stopping and  repeating the cryovacuum test, this reserve could easily be exhausted.  Additionally, as the project moves further into integration and testing,  events become more serial so flexibility will be diminished. Issues  uncovered in integration and testing also tend to be more expensive to  mitigate, due to increased schedule pressure.
To prevent the use of additional schedule reserve, the project and its  contractor for OTIS testing are taking proactive steps to reduce risk  before testing needs to commence by ensuring the availability and  readiness of test equipment and the cryogenic chamber to be used to test  the optics of JWST. For example, the project’s contractor that is to test  the optics has conducted two of three optical ground support equipment  tests on a replica of the OTE with 2 of 18 primary mirror segments  installed. According to the project, the first test met its intended objectives  and provided valuable insight into the performance of the ground support  equipment and preparation of the cryogenic chamber. The second test  was completed in October 2015, and project officials are currently  analyzing the results. The third test is to build upon these findings to  provide further confidence for the eventual OTIS testing. Additionally, the  contractor performed several risk mitigation activities, including additional  testing of the large cryogenic chamber that will be used for OTIS testing,  which revealed several issues, including a leak in the cryogenic chamber  that would have had major impacts if not discovered and repaired before  OTIS testing began.
section_title: Schedule Reserve Continues to Be Used to Address Technical Challenges
paragraphs: The project has used schedule reserve in 2015 to address various  technical problems that have arisen. More specifically, the project  experienced several problems with ISIM and OTE, elements in the two of  five integration and test phases that have begun. For example, the ISIM  heat straps—flexible straps that are to conduct energy and heat away  from the instruments—did not perform as expected in testing. An  investigation revealed design issues with the parts as delivered from the  supplier. As a result, the heat straps were redesigned and reinstalled,  which required the use of schedule reserve. Additionally, as a result of  these and other issues, the beginning of the third cryovacuum test was  delayed by 3 months. ISIM currently holds 1.75 months in schedule  reserve—down from 4.5 months as we found last year—from its overall  schedule reserve of 8.75 months to address any issues that may arise  during the third cryovacuum test and before OTIS testing begins.
Additionally, the OTE element used about 2 months of schedule reserve  this past year due to workmanship issues related to the 76 cryogenic  harnesses that connect to JWST’s mirrors. According to program and  contractor officials, the majority of these harnesses were damaged due to  use of inappropriate tooling by the supplier. The damage was not  discovered until some of the harnesses were installed on the OTE. The  harnesses were removed for inspection with most requiring repairs or  replacement. According to contractor officials, initially, the harnesses  would have been installed at Northrop Grumman’s facility in Redondo  Beach, California, but due to the workmanship issues, and in an effort to  preserve as much schedule as possible, all but two of the harnesses are  being installed at Goddard after the OTE was transferred there to begin  the integration of the mirrors with the backplane.
Various spacecraft challenges during the past year have used about 3  months of schedule reserve. For example, Northrop Grumman planned  for certain integration activities to be conducted concurrently. However,  according to project officials, due to safety and access to the spacecraft  bus, the work had to be completed sequentially instead which took longer  than expected. Additionally, a propellant tank required redesign and  rework to meet its requirements. Spacecraft bus structure integration has  been completed and the bus assembly recently completed various fit  checks and acoustics and dynamics testing in preparation for the  spacecraft integration and testing phase to begin in 2016.
Schedule reserve for the sunshield was reduced to 9.25 months—two  weeks from the critical path—due to various manufacturing challenges,  and additional reserves will likely be needed in the near future. For  example, an anomaly with the membrane retention devices—which need  to operate correctly to ensure that the sunshield can unfold properly— during qualification testing required a redesign of the parts. According to  contractor officials, when the devices were released, the contact between  the metal surfaces moving adjacent to one another resulted in a small  amount of debris being generated. Project officials expressed concern  that the debris posed a risk of damaging other parts of the telescope. A  new design has since been tested and proven to no longer pose the  same risk. Additionally, coordinating the testing of the five individual  layers of the sunshield created some delays. The five layers of the  sunshield are currently in various stages of assembly, with two layers  having been delivered to Northrop Grumman from the supplier in April  2015 and November 2015, respectively. In addition, in October 2015, the  project reported that a piece of flight hardware for the sunshield’s mid  boom assembly was irreparably damaged during vacuum sealing in  preparation for shipping. The effect of the accident on the schedule has  not yet been determined as project and prime contractor officials are  currently determining the path forward.
section_title: Cryocooler Delivered but Remains a Significant Schedule Risk during Testing
paragraphs: The cryocooler continued to experience technical challenges in 2015 that  used schedule reserve and delayed its delivery. Although it has now been  delivered—approximately 18 months later than planned—the cryocooler  remains a schedule risk as it begins testing. Northrop Grumman delivered  the compressor assembly—the third and final cryocooler component to be  delivered after the cold head assembly and electronics assembly—to JPL  in July 2015. Over the last several years, the project has accommodated  a series of cryocooler schedule slips by reordering and compressing  JPL’s test schedule and resequencing the spacecraft bus integration  schedule. For example, several tests that were initially planned to be  conducted on flight hardware will now be conducted on the spare  hardware later into JWST’s integration and testing phase and closer to  launch. Additionally, in May 2015, the project used 3 weeks of reserve  and removed 3 weeks of lower priority and redundant items from the  planned 40 weeks of acceptance and end-to-end testing. The project  took these actions to accommodate a further delay in the delivery of the  compressor assembly. According to contractor officials, the delay was  primarily caused by the contractor not scheduling enough time to  complete the bake out—a process whereby moisture is removed by  heating the compressor and pumping helium through it. Table 1 below  shows the tests removed from the acceptance and end-to-end testing.
According to JPL officials, the thermal tests removed were from those that  would have tested the cryocooler compressor assembly with the  electronics assembly. A program official stated that the risk of eliminating  these tests has been reduced now that the compressor and electronics  assembly have been tested together for the first time. NASA and JPL  assessed the removal of the electromagnetic interference and  electromagnetic compatibility testing as low risk, and a program official  stated that additional parallel activities and testing of the spare electronics  to further mitigate the risk have been added. However, various integration  and test experts we spoke with noted that eliminating testing is a sign that  the project may be taking on additional risk because discovery of issues  may be pushed to higher levels of testing or to later in the testing phase,  where problems are more costly and time consuming to address.
To accommodate any delays to testing or problems that may be found,  JPL currently maintains 12 weeks of reserve for acceptance and end-to- end testing of the cryocooler. According to JPL officials, a key driver in  deciding to eliminate testing instead of using additional schedule reserve  was to retain as much as possible in the event that a test has to be  stopped and restarted—which would require approximately 5 weeks—in  addition to the time it takes to mitigate a problem. At the completion of the  acceptance and end-to-end testing programs, the cryocooler is needed  for spacecraft integration and testing—when the spacecraft and sunshield  are integrated—no later than August 2, 2016. Spacecraft integration and  test is followed by the final observatory level integration and test— completing the telescope—which is expected to begin in September  2017. The cryocooler’s testing flow and schedule reserve leading to its  integration with the spacecraft is depicted in figure 5 below.
Because the development and delivery of the cryocooler by Northrop  Grumman took significantly longer than expected and to maintain the 12  weeks of reserve, JPL must complete acceptance and end-to-end testing  in a more schedule-compressed environment. However, challenges have  persisted in bringing the cryocooler flight model to testing and completing  development of the spare model which could be needed if the flight model  is not available in time to be integrated into the observatory for launch.  For example, despite having an extra 18 months to prepare for the  cryocooler testing due to the delay in the delivery of the compressor  assembly, the project noted concerns with JPL’s readiness to accept the  flight hardware. Specifically, a procedure error led to an interruption of the  ongoing testing of the electronics assembly and fit checks of the flight  cooler tower assembly and flight refrigerant line deployable assembly  were delayed because the procedures were late in being completed.
section_title: Unusual Complexity of the JWST Project Heightens the Importance of Integration and Testing Phase to Reduce Substantial Risks
paragraphs: JWST is one of the most technologically complex projects NASA has  undertaken. The project incorporates nine critical technologies— technologies that are required for the project to successfully meet  requirements—whereas we found the average technology development  project at NASA incorporates an average of 2.3 critical technologies.  JWST also incorporates 15 pre-existing technologies that are being  leveraged from previous development efforts.
Future testing on JWST has to reduce a significant amount of risk before  the October 2018 launch. The project identifies and maintains a list of  risks—currently with 102 items—that need to be tested and mitigated to  an acceptable level in the next 3 years. According to the project,  approximately 25 of these risks are not likely to be closed until the  conclusion of the observatory integration and test phase—just prior to  project launch. This is the point where the project has determined that no  further mitigations are feasible and that these risks have been tested per  a plan to reduce the risk, when possible, to an acceptable level. In some  cases, it may take years to resolve a particular risk. For example, the  project continues to track a risk related to the release mechanisms that  hold the spacecraft and the OTE together for launch. Once in space, they  are to activate and release to allow the OTE to separate from the  spacecraft. If the mechanisms do not operate correctly, mission failure will  occur. This risk was identified in January 2014 at the spacecraft critical  design review. During testing, these devices were causing excessive  shock when performing their releasing function. After redesign, the project  is continuing to work on resolving the underlying problem and qualifying  the new design. Additionally, while testing the redesigned mechanism, a  new concern arose that it could release early which would cause mission  failure. According to a program official, the redesigned mechanism is not  needed for spacecraft integration and test until summer of 2016 and  therefore these issues do not pose a significant schedule concern at this  time.
As integration and testing moves forward, the project will need to be able  to resolve problems in a timely manner to stay on schedule. A backlog of  unresolved problems and risks may indicate there may not be enough  schedule left before launch to complete all necessary work. The project  keeps track of these problems via problem reports and problem failure  reports. Thus, ensuring that problem reports and problem failure reports  are resolved in a timely manner is key to successfully launching the  project on time. Project officials reported that they do not track problem  report and problem failure report closure rates over time; instead, they  monitor the reports to ensure that they are closed before subsequent test  events and receive monthly briefings from the contractors on the status of  their progress. According to project officials responsible for the ISIM and  OTIS development and testing, while there are numerous problem reports  open at any given time, they are comfortable with the number of open  reports at this stage of the project. Northrop Grumman officials reported  that development of the OTE, sunshield, and spacecraft are on track with  respect to problem and failure reports, which they refer to as  nonconformance reports. Additionally, experts we spoke with told us that  addressing requests for action (RFAs) from project reviews is important  because RFAs are written to identify potential risks to the project. Since  the spacecraft critical design review in January 2014, the project has  closed 14 RFAs from that review while one related to the release  mechanism noted above remains open. The project tracks and reports  open RFAs to senior management at NASA, and we will continue to  examine the open RFA and additional RFAs that result from future  reviews to monitor their timely closure.
The extent of JWST’s deployments—which are necessary because JWST  must be stowed for launch to fit in the launch vehicle—means the  telescope could fail to operate as planned in an extensive number of  ways. According to project officials, there are over 100 different ways that  a failure could occur, referred to as single point failure modes, across  hundreds of individual items in the observatory. Each of these could  result in a loss of minimum mission objectives, and thus needs to be fully  tested and understood. Nearly half of the single point failure modes  involve the deployment of the sunshield. The approval of single point  failures requires written justification from the project including sound  engineering judgement, supporting risk analysis, and implementation of  measures to mitigate the risk to acceptable levels. The project’s mission  systems engineers have developed justifications and mitigation strategies  for its single point failures, and project officials expect these to be  summarized and submitted to the agency prior to launch. According to  project officials, this approach is consistent with other high-priority NASA  missions, which require the most stringent design and development  approach that NASA takes to ensure the highest level of reliability and  longevity on orbit.
section_title: Project Continues to Meet Cost Commitments but Contractor Workforce Increases and Unreliable Contractor Performance Data Pose Management Challenges
paragraphs: The JWST project continued to meet its cost commitments throughout  fiscal year 2015 despite cryocooler delays that used a disproportionate  amount of cost reserves. However, the project required larger than  planned workforce levels to complete new and existing work, which poses  a cost threat in future years if levels do not decrease. To help manage the  project and account for new risks since the 2011 replan, JWST project  officials conducted a cost risk analysis of the Northrop Grumman contract.  We found that while the cost risk analysis substantially met best  practices, these officials do not plan to periodically update it. Instead, the  project is using risk-adjusted analyses to update and inform its cost  position. However, we found that this method is a simplified version of a  cost-risk analysis that does not contain the same rigor or allow the project  to prioritize risks. Furthermore, we found anomalies in the contractor- provided data rendering the results of the analyses unreliable. Finally, we  also found the project lacks an independent surveillance mechanism for  the data to ensure anomalies are corrected by the contractor before being  incorporated into larger analyses. As a result, the project is relying  partially on unreliable information to inform its cost and schedule  decision-making.
subsections: section_title: Project Continues to Stay on Budget and Spend within Cost Commitment Despite Significant Cryocooler Cost Growth
paragraphs: Project officials managed JWST within its allocated budget for the fourth  consecutive year since the 2011 replan. Additionally, the project’s fiscal  year 2016 budget request to Congress is consistent with its cost  commitment. According to preliminary estimates, at the end of fiscal year  2015, the project spent $68 million dollars more than planned at the  beginning of the fiscal year, carrying over less money into fiscal year  2016 than originally planned. As in past years, the project used a portion  of its cost reserves to address technical challenges that included funding  activities to address significant delays with the cryocooler. The project  also used program-level cost reserves to pay for new work that included  conducting additional thermal verification tests and risk reduction  activities, such as an analysis to better understand how JWST will likely  interact with its launch vehicle—the Ariane 5.
The cryocooler used a significant share of the project’s fiscal year 2015  cost reserves—more than 50 percent—to fund the workforce for this effort  and address technical issues. This is the fourth year in a row that the  cryocooler used a substantive portion of the project’s cost reserves to  further fund the subcontractor’s schedule delays in delivering its  components. The project estimates that the overall cryocooler  development cost will be nearly 250 percent higher than baselined at the  2011 replan. The Northrop Grumman cryocooler team forecasts that a  larger workforce is needed until at least February 2016 when the spare  compressor assembly is currently scheduled to be delivered. JPL will  maintain the majority of its workforce through the conclusion of spare  cryocooler testing. After testing concludes, its workforce is projected to  decrease by about 50 percent. Project cost reserves will likely continue to  be needed to fund cryocooler development and testing costs until fiscal  year 2017 when JPL testing of the spare compressor assembly is  scheduled to conclude.
section_title: Larger Than Planned Workforce Poses a Threat to Meeting Cost Commitment
paragraphs: While the project remains on cost, contractor work is costing more to  complete because a larger workforce than planned was needed for  components beyond the cryocooler, including Northrop Grumman for the  sunshield, spacecraft, and OTE, and Harris for OTIS testing and  preparation. This need derives from work taking longer than planned to  complete and additional work requested by NASA. For example, Northrop  Grumman’s workforce projections for fiscal year 2015 predicted a peak in  the workforce in November 2014. However, the actual workforce peaked  in February 2015 and continued to remain above the projected peak until  August 2015. While workforce numbers have declined somewhat since  February, these increases largely remained in place through the end of  the fiscal year. In its role as prime contractor, Northrop Grumman’s  workforce stayed within its budget in fiscal year 2015. From January  through July of 2015, its workforce was exactly at its funding threshold in  order to conduct new work and address technical issues for its body of  work. In addition, larger workforces contributed to additional contractor  cost for two other development efforts—OTIS testing and the  cryocooler—requiring the use of additional project cost reserves.
Looking forward, the primary threat to JWST meeting its long-term cost  commitment is the prime contractor, which must continue to control its  costs and decrease its workforce. For the past 20 months, Northrop  Grumman’s actual workforce exceeded its projections. Figure 6 below  illustrates the difference between the workforce levels that Northrop  Grumman projected at the beginning of fiscal years 2014 and 2015 and  its actual workforce levels for those periods.
Based on its projections at the beginning of the fiscal year, Northrop  Grumman exceeded its total fiscal year 2014 workforce monthly  projections by about 12 percent, and exceeded its projections for fiscal  year 2015 by about 20 percent. On average, in fiscal year 2015, Northrop  Grumman was 121 FTEs above its projections each month, and at the  end of fiscal year 2015, it exceeded its monthly projection for September  2015 by 235 FTEs. While actuals have remained above projections since  the workforce levels peaked in February 2015, Northrop Grumman  currently projects that its workforce will decline throughout fiscal year  2016, with the exception of August 2016, when additional work is  projected to be needed for integration and testing, among other areas.  However, this was the projection for both fiscal years 2014 and 2015 and  has yet to happen. For example, while Northrop Grumman expected to be  ramping down by the end of fiscal year 2014, its projections at the start of  fiscal year 2015 were approximately 55 percent higher than where  workforce levels were projected for the end of fiscal year 2014. The  primary drivers that have increased the cost and size of the workforce  under the prime contract have been the development of the sunshield and  spacecraft and additional work NASA has requested. Over 60 percent of  the cost increases are attributed to addressing technical concerns such  as sunshield alignment and verification work, mechanical design  integration, and spacecraft mass reduction. Northrop Grumman has  covered additional costs pertaining to technical issues through its  management reserves, and has not needed project cost reserves in fiscal  years 2014 and 2015. The remaining cost increases are attributable to  new contract scope which has been funded by the JWST program. Some  of this new scope included additional spacecraft simulators, as well as  new thermal risk reduction testing to verify the final design changes made  to the core of the telescope–the region between all the observatory  elements.
Approximately 15 percent of work remains on Northrop Grumman’s  contract and its management reserves exceed the recommended  minimum amount that should be held at the contractor level—10 percent  or more of the cost of work remaining on the project. Significant  decreases in the workforce are planned to occur in fiscal year 2017 when  final hardware delivery to observatory integration and test is scheduled to  take place. To incentivize the contractor to lower its workforce, project  officials evaluate workforce management as part of NASA’s appraisal of  Northrop Grumman’s performance in its award fee determinations. The  project also communicates frequently with the contractor including phone  calls, face to face meetings twice a month, and quarterly in-person  management meetings to discuss workforce planning, among other  subjects. The project has communicated the need to reduce the  workforce size, but since Northrop Grumman has operated within its  budget in fiscal years 2014 and 2015, the award fee it has received has  not been reduced as a result of workforce size issues.
The subcontractor for OTIS testing, Harris, needed additional funding to  cover cost overruns and additional work. Project cost reserves were  utilized to pay for this work to maintain schedule through a contract  change in January 2015. Over 55 percent of the increase was made to  address cost overruns that resulted from increasing workforce levels to  maintain schedule. The rest of the contract increase covered new scope.  As a result, Harris is anticipating more work than originally planned for  fiscal years 2016 and 2017. Despite the contract increases, Harris’s  management reserves are 2.5 percent as of August 2015—significantly  below the 10 percent cost of work remaining that is considered to be  healthy. With over 25 percent of work remaining on its contract, this low  level of reserves means that any additional overruns will likely need to be  covered by project-held reserves.
section_title: NASA’s JWST 2014 Cost Risk Analysis Substantially Met Best Practices
paragraphs: We found that NASA’s 2014 cost risk analysis on Northrop Grumman’s  remaining work substantially met best practices. In December 2014, we  recommended that project officials update the 2011 JWST cost risk  analysis utilizing best practices, and to update it periodically as significant  risks emerge. NASA partially concurred with our recommendation  stating that the program and project use a range of tools to assess the  performance of the project and conducted a one-time update to the cost  risk analysis in 2014. We found that NASA’s updated cost risk analysis  substantially met best practices. For example, it incorporated subject  matter expert input to model cost and schedule uncertainties from the  prime contractor’s threats and opportunities list—both of which are  components of the best practice of modeling a probability distribution for  each cost element’s uncertainty based on data availability, reliability and  variability. See appendix I for a list of best practices that we used to  evaluate cost risk and uncertainty. In addition, NASA included correlation  between elements to account for different cost elements being affected by  the same external factors—another best practice.
However, the cost risk analysis also had some weaknesses as a result of  not fully following best practices. For the first best practice noted above  on modeling probability distribution, NASA relied on the contractor’s risk  data without conducting corroborating interviews with contractor  personnel to obtain insight into threats and opportunities not listed in  contractor data. For the same best practice, the detailed schedule that  reflected all of the work that needed to be done by Northrop Grumman  that was used for the cost risk analysis had some activity sequencing  logic issues. For example, we found instances where activities listed were  not sequentially linked to one another. As a result, this called into  question the calculation of the critical path during simulations as well as  the ability of the schedule to dynamically respond to changes, which it  must do thousands of times during the risk simulations. Moreover, the  JWST project does not plan to periodically update its cost risk analysis  even as additional risks have emerged. JWST officials stated that the cost  risk analysis was a time intensive process to complete and that the  program and project use various tools consistent with best practices to  assess all major contractors’ performance. Nonetheless, best practices  call for conducting periodic updates to a cost risk analysis as a project  progresses even if it is not experiencing problems. Updating the cost risk  analysis is also part of the best practice of implementing a risk  management plan with the contractor which calls for identifying and  analyzing risk, planning for risk mitigation, and continually tracking risks.  An accurate cost risk analysis is particularly vital to JWST because about  70 percent of the project cost reserves have been used to address  concerns that were not anticipated as threats by the project’s budget  system. Failure to update the cost risk analysis as we recommended in  2014 limits stakeholder confidence that the cost risk analysis prepared in  2014 accurately reflects the project’s current financial status. Given this  uncertainty, it is important for the project to have reliable information for  the risks that are known to inform decision making.
section_title: Unreliable EVM Data Limit Project’s Ability to Reliably Forecast Future Cost Growth for Contractors
paragraphs: One of the tools that the project has started to use in place of updating  the cost risk analysis is a monthly risk-adjusted analysis to provide insight  into potential future cost growth. The monthly risk-adjusted analyses are  based on contractor EVM data that incorporate known threats to provide  an estimate at completion (EAC) that is updated monthly by NASA for  each contractor. The results of these analyses may then be compared to  the contractors’ estimates and project cost reserves to provide insight into  the project’s ability to cover future increases. Monthly risk-adjusted  analyses demonstrate a commitment by NASA to manage and project  future costs.
However, we found that the risk-adjusted analyses do not serve as an  adequate substitute for an updated cost risk analysis because they are a  simplified version of a cost risk analysis that does not allow the project to  prioritize risks or assign confidence levels to meet key milestones in the  schedule consistent with best practices for cost risk analyses.  Additionally, based on our analysis of contractor EVM data over 17  months, we found that some of the data used to conduct the analyses  were unreliable. First, we found that both Northrop Grumman and Harris  were reporting optimistic EACs at the time of our analysis that did not  align with their historical EVM performance and fell outside the low end of  our independent EAC range. Second, we found various anomalies in  contractor EVM data for both the Northrop Grumman and Harris work that  they had not identified throughout the 17-month period we examined. The  anomalies included unexplained entries for negative values of work  performed (meaning that work was unaccomplished or taken away rather  than accomplished during the reporting period), work tasks performed but  not scheduled, or actual costs incurred with no work performed. For  Northrop Grumman, many were relatively small in value ranging from a  few thousand to tens of thousands of dollars. These anomalies are  problematic because they distort the EVM data, which affect the  projection of realistic EACs. We found that these anomalies occurred  consistently within the data over a 17-month period, which brings into  question the reliability of the risk-adjusted EAC analysis built upon this  information. NASA did not provide explanations into the anomalies for  either contractor. While the contractors were able to provide explanations  for the anomalies upon request, their explanations or corrections were not  always documented within EVM records. Some of the reasons the  contractors cited that were not in the EVM records included tasks  completed later than planned, schedule recovered on behind schedule  tasks, and replanning of customer-driven tasks. Finally, like the cost-risk  analysis in 2014, the risk-adjusted EAC analysis does not include  interviews with contractor officials to gain insight into risks which may not  be present in the contractors’ threats and opportunities list. Without  updating the cost risk analysis, reconciling and documenting data  anomalies, and utilizing reliable data for the risk-adjusted EAC, the JWST  project does not have a reliable method to assess its cost reserve status  going forward. This means that some of the cost information the project  officials use to inform their decision making may indicate they are in good  shape when the reality might be otherwise, and as result, project  management may not have a solid basis for decision making.
In discussions with the contractors, we found that the project also lacks  an independent surveillance mechanism, such as the Defense Contract  Management Agency, to monitor contractors’ EVM data—provided to the  project each month from two of the contractors. Surveillance entails  reviewing a contractor’s EVM system with the purpose of focusing on how  well a contractor is using its EVM system to manage cost, schedule, and  technical performance. However, the lack of surveillance and the data  anomalies in EVM data are problems we previously identified across  NASA’s portfolio of major spaceflight projects. We found in November  2012 that 4 of NASA’s 10 major spaceflight projects we reviewed had  established formal independent surveillance reviews. For the 6 projects  that did not have formal independent surveillance in place, we found that  each provided evidence that they instituted monthly EVM data reviews,  which according to project officials, helped them to continually monitor  cost and schedule performance. However, we found that the rigor of both  the formal and informal surveillance reviews was questionable given the  numerous EVM data anomalies we found in the monthly EVM data. As a  result, we recommended that NASA improve the reliability of project EVM  data by requiring projects to implement a formal surveillance program that  ensured anomalies in contractor-delivered and in-house monthly earned  value management reports were identified and explained, and report  periodically to the center and mission directorate’s leadership on relevant  trends in the number of unexplained anomalies. Citing resource  constraints, NASA partially concurred with the recommendation and  commented that it did not plan to implement a formal surveillance  program, but agreed that the reliability and utility of the EVM data needed  to be improved and noted several steps it planned to take to do so. We  continue to believe that implementing this recommendation would be  beneficial and prevent anomalies in EVM data from occurring that we  have identified on the JWST project. Implementing surveillance of EVM  contractor data is a best practice listed in the NASA Earned Value  Management Implementation Handbook and GAO’s Cost Estimating and  Assessment Guide. With adequate surveillance in place, the anomalies  we found in the EVM data could have been identified earlier and  corrective action could have been directed to the contractors to explain  the anomalies in the data. Without implementing proper surveillance, the  project may be utilizing unreliable EVM data in its analyses to inform its  cost and schedule decision making.
section_title: Independent Oversight Has Been Built into JWST’s Development Life Cycle to Improve the Likelihood of Mission Success
paragraphs: NASA has taken steps to provide independent oversight of the JWST  project. Independent oversight of the JWST project has played and will  likely continue to play an important role leading up to JWST’s launch in  October 2018. Before the 2011 replan, two groups examined JWST to  address underlying concerns with schedule and cost and made  recommendations that NASA implemented. On an ongoing basis until  launch, the Standing Review Board and the Independent Verification &  Validation (IV&V) facility are to continue to oversee progress on hardware  and software development, identify concerns, and assist the project to  identify solutions to reduce risk and improve JWST’s likelihood of  success.
subsections: section_title: Independent Oversight Has Provided Insight to Reduce Risk
paragraphs: Various groups internal and external to NASA have conducted reviews,  provided insights, and identified schedule efficiencies to inform and  enhance the project’s approach to managing the development of JWST.  Prior to the 2011 replan and because of concerns raised at the JWST  mission critical design review held in the spring of 2010, the Test  Assessment Team was formed to address those concerns. Convened by  the Astrophysics division of the NASA Science Mission Directorate, the  team included nine members and three NASA consultants with  considerable experience in systems engineering, instrument  development, system verification, modeling and testing, and other areas  focused on reviewing plans for the ISIM and OTIS cryogenic testing. The  team was primarily tasked to determine whether (1) the test plans in place  at that time were sufficient to test the relevant observatory functions, (2)  the key optical and thermal objectives were clearly identified, (3) the test  plans themselves were properly scoped and prioritized, (4) any  duplicative or unnecessary tests existed in the plans, and (5) the plans  were overly ambitious or optimistic regarding hardware performance and  analysis capabilities. Their insights and recommendations have helped to  decrease programmatic cost and future growth as well as to find schedule  efficiencies. For example, they recommended OTIS testing duration be  reduced from 167 to 90 days while still verifying critical functions of the  telescope. Also prior to the 2011 replan, the Chair of the Senate  Subcommittee on Commerce, Justice, Science, and Related Agencies  asked that NASA set up a panel to review the JWST project because of  concerns about cost growth and schedule delays. In response, NASA  convened the Independent Comprehensive Review Panel to provide an  independent, integrated perspective and response with the goal of  providing recommendations that would lead to a successful launch while  minimizing cost. At the conclusion of its work in October 2010, the panel  made 22 recommendations to NASA to increase oversight, improve  communications, and assist with risk management and mitigation, among  other recommendations. NASA implemented all of these  recommendations. Both of these reports have informed our ongoing  reviews of the JWST project as we have incorporated many of the  concerns on cost estimates and cost reserves into our methodology and  reporting on the health and status of JWST as it moves forward.
section_title: Independent Standing Review Board Provides Additional Expertise and Insight to Help Manage Project
paragraphs: Another aspect of independent oversight that is a key element of NASA’s  strategic framework for managing space flight projects are Standing  Review Boards which consist of technical experts who do not actively  work on a specific project or program. The mission of the boards is to  provide NASA senior management with objective information to ensure  there is appropriate program and project management oversight to  increase the likelihood of mission success. The boards help to determine  the adequacy of programs’ (1) management approach, (2) technical  approach, (3) integrated cost and schedule estimates and funding  strategy, and (4) risk management, among others. NASA’s Independent  Program Assessment Office and various NASA centers organize these  boards and coordinate their involvement at different reviews. The  boards are involved at various agency-level reviews with some members  participating in lower-level reviews at NASA’s different centers, in monthly  reviews held by the projects and program, or in special reviews on a  specific topic or set of issues. Standing Review Boards may also make  non-binding recommendations after life-cycle reviews, but do not have  programmatic or technical authority over the programs or projects. The  Standing Review Board Handbook describes three types of boards that  may be formed to provide independent oversight of programs or projects.  See table 2 below for the three types of Standing Review Boards.
NASA’s Standing Review Board Handbook states that a civil servant  consensus with no expert support is the preferred structure within NASA  because experience demonstrates that a consensus board leads to a  more meaningful discussion of the review findings and recommendations,  especially where dissenting opinions are discussed. A non-consensus  mixed board provides only the perspective of the chairman. In 2015, 1 of  33 active Standing Review Boards was a civil service consensus board  with no expert support, 15 were civil service consensus with consultant  support, and 17 were non-consensus mixed boards. Although NASA  guidance prefers civil servant consensus boards, NASA officials told us  that they have found it challenging to staff boards exclusively with civil  servants for a number of reasons including availability of staff, finding a  person with the appropriate skill set, and independence reasons, among  others.
JWST has had a number of changes occur on the boards overseeing the  project for different reasons. JWST has had a Standing Review Board  since 2006 when a special review was conducted. During that review and  from 2008 to 2014, the board was a non-consensus board led by an  outside expert chosen by NASA senior officials. The experts were civil  servants as well as non-civil-servant experts. In May 2014, the chairman  retired, a new chair was appointed the same year, and NASA senior  officials changed the board to a consensus board with consultants.  Independent Program Assessment Office officials told us that board types  can change for numerous reasons, including when a project or program  enters a different phase of development that may require different  technical skills or if all of the convening authorities request it. As a result  of the retirement of the chairman, most of the 2008-2014 Standing  Review Board members who were not civil servants but who had  overseen JWST for more than 6 years were replaced and 2 civil servants  were carried over to the new board. Consultant support was added for  schedule analysis and in one technical area to support launch vehicle  integration because NASA has never launched a mission on an Ariane 5  rocket as it plans to do for JWST. Before retiring in 2015, the previous  Standing Review Board chairman expressed the importance of having  representation from JPL as a member of the board to provide experience  working on unmanned spacecraft projects—but a JPL member could not  be added since JPL employees are not civil servants and can only be  consultants to the board. With the appointment of a new chairman in  October 2015, there have been additional membership changes to the  board including the addition of a JPL consultant.
section_title: Independent Oversight of Critical Software Confirms Development Is on Schedule with Concerns Regarding Large Number of Developers
paragraphs: NASA’s IV&V facility—which independently examines software  development—reviews mission critical software for most NASA programs  and projects to achieve the highest levels of safety and cost-effectiveness  by ensuring that developed software will perform as required. Experts at  the facility work to uncover high-risk errors early in the development life  cycle of software for many NASA programs and projects. IV&V is a  process whereby organizations can reduce the risks inherent in system  development and acquisition efforts by having a knowledgeable party who  is independent of the developer to determine whether the system or  product meets the users’ needs and fulfills its intended purpose. IV&V  applies software engineering best practices to risk elements on safety  critical and mission-critical software throughout the development life  cycle. We have found IV&V to be a leading practice for federal agencies  in managing their complex, large-scale, or high-risk acquisition of  programs. Software development is a challenge we have found on  many different acquisitions—some space-related—in government  programs that has led to schedule delays and cost growth. Examples  include the F-35 Joint Strike Fighter, the Aegis Modernized Weapon  System, NASA’s Stratospheric Observatory for Infrared Astronomy, and  Geostationary Weather Satellite development, among others.
The goal of IV&V is to examine the three following questions regarding  software:
Will the system do what it is supposed to do?
Will the system not do what it is not supposed to do?
Will the system perform as expected under adverse conditions?
IV&V is required to examine software on all projects with a life cycle cost  over $1 billion, other projects over $250 million with a higher risk  classification, or those specifically selected by the NASA Chief, Safety  and Mission Assurance. Once selected, a portfolio-based risk  assessment is developed to identify top-level mission capabilities and a  risk based assessment process identifies the most important system  capabilities and the software components that play the most important  role in the mission. IV&V officials noted that due to limited resources, they  examine mission and safety critical software and they do not have the  funding to examine all programs or projects across NASA’s portfolio.
Generally, IV&V officials stated that they believe JWST’s software  development is going well, but the testing that lies ahead—when the  different components are integrated—will be a challenge. For example,  IV&V officials noted that JWST’s software build is the largest they have  reviewed for a science mission, but not the largest they have reviewed  across NASA as some Human Exploration Operations are larger. They  said that most of JWST’s software required to position and deploy the  telescope will be examined by IV&V. However, they noted that JWST’s  integration is more challenging, primarily due to the number of software  developers involved. While most science programs or projects have two  to four software developers, JWST has eight. JWST’s software  development has been examined by IV&V since fiscal year 2004 and,  according to officials, will likely continue to be examined until after launch  when operations begin. IV&V officials said they do not examine the  software for the launch vehicle.
While IV&V’s function requires independence from programs and  projects, there have been recent changes in funding that have reduced its  financial independence to some extent. Organizationally, the IV&V Facility  remains independent by reporting to the Office of the Director of Goddard  and the Office of Safety and Mission Assurance at NASA Headquarters— not to the programs or projects it examines. However, financially, starting  in 2015, an IV&V financial management official said that 75 percent of the  IV&V’s funding came from NASA Headquarters via the Agency  Management Operations fund and the remaining 25 percent was divided  amongst the various mission directorates. This changed from the past 10  years, when 100 percent of the IV&V Facility’s budget came from the  Agency Management Operations to ensure the independence of the IV&V  office. We have previously found that financial independence requires  that the funding for IV&V be controlled by an organization separate from  the development organization. This ensures that the effort will not be  curtailed by having its funding diverted to other program needs, and that  financial pressures cannot be used to influence the effort. As a user of  IV&V’s expertise, JWST, via the use of program cost reserves,  contributed a small portion of funding to the software IV&V facility to help  fund their budget in fiscal year 2015. While this financial situation was  new in fiscal year 2015, we will continue to monitor how NASA deals with  funding the IV&V facility in the future to protect its independence.
section_title: Conclusions
paragraphs: The JWST project has made progress building, integrating, and testing  significant portions of JWST in the past year within the commitments  made at the time of the 2011 replan for both schedule and cost. With the  third major integration and test period starting in 2016, resolving technical  challenges in a timely manner, and ensuring the OTIS test goes smoothly  are key to continued progress within the project’s schedule commitment.  Additionally, reducing the size of Northrop Grumman’s workforce and  controlling costs within the fiscal year 2016 budget will be key metrics to  monitor to demonstrate the project can meet requirements within its cost  commitment. In the past, the project has benefited from independent  expertise, information, and recommendations to improve the  management of the project.
Moving forward, the project may benefit from having more reliable data  provided from its contractors to ensure that its EACs, which take into  account risks and threats, are better able to inform its cost status. While  the contractors were able to explain the anomalies, most had not been  previously identified or documented. NASA used the data for its analyses,  which subsequently raised questions about the reliability of those  analyses. Making management decisions using unreliable data can result  in bad decision making and can misinform the project on its long-term  financial position which may have significant consequences if not  corrected. We recommended in our December 2014 report that NASA  conduct a cost risk analysis and follow best practices, which include  updating it as risks change during the life of the program. Because the  project is not going to conduct another cost risk analysis, putting  independent surveillance in place to improve the accuracy of its risk- adjusted analysis—despite its weaknesses relative to the information a  cost risk analysis provides—will provide better information to inform its  decision making. In November 2012, we recommended that NASA  improve the reliability of project EVM data by requiring its major  spaceflight projects to implement a formal surveillance program that  ensured anomalies in contractor-delivered data and in-house monthly  EVM reports were identified and explained. NASA partially concurred with  this recommendation but has not taken steps to require surveillance on  projects like JWST. However, we continue to believe that improving the  surveillance on projects will help reduce data anomalies from occurring  like the ones we identified on JWST, resulting in better information and  analyses to inform project decision making.
section_title: Recommendation for Executive Action
paragraphs: To resolve contractor data reliability issues and ensure that the project  obtains reliable data to inform its analyses and overall cost position, we  recommend that the NASA Administrator direct JWST project officials to  require the contractors to identify, explain, and document all anomalies in  contractor-delivered monthly earned value management reports.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to NASA for comment. In written  comments, NASA agreed with our recommendation. These comments  are reprinted in appendix IV. NASA also provided technical comments,  which have been addressed in the report, as appropriate.
We are sending copies of the report to NASA’s Administrator and  interested congressional committees. In addition, the report will be  available at no charge on GAO’s website at http://www.gao.gov.
Should you or your staff have any questions on matters discussed in this  report, please contact me at (202) 512-4841 or chaplainc@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to assess (1) the extent to which technical  challenges have impacted the James Webb Space Telescope (JWST)  project’s ability to meet its schedule commitments, (2) the current cost  status of the JWST project and the primary challenges that may influence  the project’s ability to meet its future cost commitments, and (3) the extent  to which independent oversight provides insight about project risks to  management.
To assess the extent to which technical challenges have impacted the  JWST project’s ability to meet its schedule commitments, we reviewed  project and contractor schedule documentation, and held interviews with  program, project, and contractor officials on the progress made and  challenges faced building the different components of the telescope. We  examined and analyzed monthly JWST project status reports to  management to monitor schedule reserve levels and usage and potential  risks and technical challenges that may impact the project’s schedule,  and to gain insights on the project’s progress since our last report in  December 2014. Further, we attended flight program reviews at the  National Aeronautics and Space Administration (NASA) headquarters on  a quarterly basis where the current status of the program was briefed to  NASA headquarters officials outside of the project. We examined  selected individual risks for elements and major subsystems from monthly  risk registers prepared by the project to understand the likelihood of  occurrence and impacts to the schedule based on steps the project is  taking to mitigate the risks. We examined previous and current test  schedules and plans to understand the sequence, what risks will be  mitigated, which risks will continue, and any reductions to planned testing.  Furthermore, we interviewed experts within and outside of NASA to  identify criteria, best practices, and metrics that may be used to assess  the project’s progress in reducing risk or provide insight into the health of  the project. Finally, we interviewed project officials at Goddard, contractor  officials from the Harris Corporation, the Jet Propulsion Laboratory, the  Space Telescope Science Institute, and different divisions of Northrop  Grumman Aerospace Systems concerning technological challenges that  have had an impact on schedule, and the project’s and contractor’s plans  to address these challenges.
To assess the current cost status of the JWST project and the primary  challenges that may influence the project’s ability to meet its future cost  commitments, we reviewed and analyzed program, project, contractor,  and subcontractor data and documentation and held interviews with  officials from these organizations. We reviewed JWST project status  reports on cost issues to determine the risks that could impact cost. We  analyzed contractor and subcontractor’s workforce plans against  workforce actuals to determine whether contractors’ are meeting their  workforce plans. We monitored and analyzed the status of program, and  project cost reserves in current and future fiscal years to determine the  project’s financial posture. We evaluated the cost risk analysis conducted  by NASA of the remaining Northrop Grumman work to determine the  extent to which all applicable best practices from GAO’s Cost Estimating  and Assessment Guide were used to build the analysis. Those best  practices included the following:
A probability distribution modeled each cost element’s uncertainty  based on data availability, reliability, and variability.
The correlation between cost elements was accounted for to capture  risk.
A Monte Carlo simulation model was used to develop a distribution of  total possible costs and an S curve showing alternative cost estimate  probabilities.
The probability associated with the point estimate was identified.
Contingency reserves were recommended for achieving the desired  confidence level.
The risk-adjusted cost estimate was allocated, phased, and converted  to then year dollars for budgeting, and high-risk elements were  identified to mitigate risks.
A risk management plan was implemented jointly with the contractor  to identify and analyze risk, plan for risk mitigation, and continually  track risks.
We examined and analyzed earned value management (EVM) data from  two of the project’s contractors to identify trends in performance, whether  tasks were completed as planned, and likely estimates at completion. We  also conducted analysis to ensure the reliability of the data over a 17- month period. In addition, we examined and analyzed risk-adjusted  analyses from NASA to determine what information they provide to the  project, the risks incorporated, their reliability, and how the project is  utilizing this information. We also discussed our assessment of the  project’s data and analysis with program and project officials to obtain  their input.
To assess the extent to which independent oversight provides insight  about project risks to management, we reviewed documentation and data  from NASA relevant groups, the program, the project, and the Standing  Review Board and held interviews with experts as well as officials from  independent oversight entities. We analyzed NASA policy and guidance  documents to understand the elements for setting up and managing a  Standing Review Board. We also reviewed the Test Assessment Team  and Independent Comprehensive Review Panel team reports to  determine how independent oversight has provided insight to JWST in the  past. We interviewed officials at NASA’s Independent Program  Assessment Office, as well as past and current Standing Review Board  members, to understand how Standing Review Boards are created,  members are selected, and how structural and personnel changes are  made over the life of NASA programs and projects, including JWST. We  also interviewed and reviewed documentation and analysis provided by  NASA’s Independent Verification and Validation group working on  JWST’s software development to determine the extent to which this group  is providing oversight of JWST software development, to determine the  health of software development on JWST, and determine what kinds of  problems remain. We did not independently review JWST’s software  development.
Our work was performed primarily at NASA headquarters in Washington,  D.C.; Goddard Space Flight Center in Greenbelt, Maryland; the  Independent Verification and Validation facility in Fairmont, West Virginia;  and by video teleconference with officials from the Independent Program  Assessment Office at Langley Research Center, Hampton, Virginia. We  also visited the Jet Propulsion Laboratory in Pasadena, California;  Northrop Grumman Aerospace Systems in Redondo Beach, California;  and the Space Telescope Science Institute in Baltimore, Maryland.
We conducted this performance audit from February 2015 to December  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Elements and Major Subsystems of the James Webb Space Telescope (JWST) Observatory
section_title: Appendix III: Organizational Chart for the James Webb Space Telescope (JWST) Program
paragraphs: Appendix III: Organizational Chart for the  James Webb Space Telescope (JWST)  Program The Jet Propulsion Laboratory is the contractor for the development of the cryocooler, but has  subcontracted most of the work to a different division of Northrop Grumman than the one that is  responsible for OTE, spacecraft, and sunshield development.
section_title: Appendix IV: Comments from the National Aeronautics and Space Administration
section_title: Appendix V: GAO Contact and Staff Acknowledgements
subsections: section_title: GAO Contact
paragraphs: Cristina Chaplain, (202) 512-4841 or chaplainc@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Shelby S. Oakley, Acting  Director; Arthur Gallegos, Assistant Director; Jay Tallon; Assistant  Director; Karen Richey, Assistant Director; Jason Lee, Assistant Director;  Brian Bothwell; Patrick Breiding; Aaron Gluck; Laura Greifner; Michael  Kaeser; Katherine Lenane; Silvia Porres; Carrie Rogers; Sylvia Schatz;  and Ozzy Trevino made key contributions to this report.