Geostationary Weather Satellites: Launch Date Nears, but Remaining Schedule Risks Need to be Addressed
Published: 2014-12-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: NOAA, with the aid of the National Aeronautics and Space Administration (NASA), is procuring the next generation of geostationary weather satellites. The GOES-R series is to replace the current series of satellites, which will likely begin to reach the end of their useful lives in 2015. This new series is considered critical to the United States' ability to maintain the continuity of satellite data required for weather forecasting through 2036.
GAO was asked to evaluate GOES-R. GAO's objectives were to (1) assess progress on program schedule, cost, and functionality; (2) assess efforts to identify and address issues discovered during integration and testing; and (3) evaluate the likelihood of a gap in satellite coverage and actions to prevent or mitigate such a gap. To do so, GAO analyzed program and contractor data, earned value data information, and defect reports, compared both defect management policies and contingency plans to best practices by leading organizations, and interviewed officials at NOAA and NASA.
section_title: What GAO Found
paragraphs: The National Oceanic and Atmospheric Administration (NOAA)'s Geostationary Operational Environmental Satellite-R (GOES-R) program has made major progress in developing its first satellite, including completing testing of satellite instruments. However, the program continues to face challenges in the areas of schedule, cost, and functionality. Specifically, the program has continued to experience delays in major milestones and cost overruns on key components. Also, in order to meet the planned launch date, the program has deferred some planned functionality until after launch, and program officials acknowledge that they may defer more.
NOAA and its contractors have implemented a defect management process as part of their overall testing approach that allows them to identify, assess, track, resolve, and report on defects. However, shortfalls remain in how defects are analyzed and reported. For example, contractors manage, track, and report defects differently due to a lack of guidance from NOAA. Without consistency among contractors, it is difficult for management to effectively prioritize and oversee defect handling. In addition, more than 800 defects in key program components remained unresolved. Until the program makes progress in addressing these defects, it may not have a complete picture of remaining issues and faces an increased risk of further delays to the GOES-R launch date.
As the GOES-R program approaches its expected launch date of March 2016, it faces a potential gap of more than a year during which an on-orbit backup satellite would not be available. This means that if an operational satellite experiences a problem, there could be a gap in GOES coverage. NOAA has improved its plan to mitigate gaps in satellite coverage. However, the revised plan does not include steps for mitigating a delayed launch, or details on potential impacts and minimum performance levels should a gap occur. Until these shortfalls are addressed, NOAA management cannot fully assess all gap mitigation strategies, which in turn could hinder the ability of meteorologists to observe and report on severe weather conditions.
section_title: What GAO Recommends
paragraphs: GAO is recommending that NOAA address shortfalls in its defect management approach, reduce the number of open high-priority defects, and add information to its satellite contingency plan. NOAA concurred with GAO's recommendations and identified steps it plans to take to implement them.

section_title: Letter
paragraphs: Geostationary environmental satellites play a critical role in our nation’s  weather forecasting. These satellites—which are managed by the  Department of Commerce’s National Oceanic and Atmospheric  Administration (NOAA)—provide information on atmospheric, oceanic,  climatic, and solar conditions that help meteorologists observe and  predict regional and local weather events. They also provide a means of  identifying the large-scale evolution of severe storms, such as forecasting  a hurricane’s path and intensity.
NOAA, through collaboration with the National Aeronautics and Space  Administration (NASA), is procuring the next generation of geostationary  weather satellites, called the Geostationary Operational Environmental  Satellite–R (GOES-R) series. The GOES-R series consists of four  satellites and is to replace the current series of geostationary  environmental satellites as they reach the end of their useful lives. This  new series is expected to provide the first major improvement in the  technology of GOES instruments since 1994 and, as such, is considered  critical to the United States’ ability to maintain the continuity of data  required for weather forecasting through the year 2036. NOAA is facing a  potential gap where it may not have a backup satellite in orbit between  now and the launch of the first GOES-R satellite. Because of the criticality  of satellite data to weather forecasting, the likelihood of data gaps in  NOAA’s satellite programs, and the potential impact of a gap on the  health and safety of the U.S. population and economy, we added  mitigating gaps in weather satellite data to our High Risk List in 2013.
This report responds to your request that we review NOAA’s GOES-R  program. Specifically, our objectives were to (1) assess progress on the  GOES-R program with respect to planned schedule, cost, and  functionality; (2) assess efforts to identify and address issues discovered  during integration and testing; and (3) evaluate the likelihood of a gap in  satellite coverage and analyze the adequacy of contingency actions in  place to prevent or mitigate such a gap.
To assess NOAA’s progress in developing GOES-R with respect to cost,  schedule, and functionality, we analyzed monthly program status briefings  to identify current status, recent development challenges, and both  expected and potential changes in functionality. We compared current  and past status briefings to determine delays over time to key milestones.  We also analyzed cost reserve and earned value data information to  understand the program’s current cost posture. To assess NOAA’s efforts  to identify and address issues discovered during integration and testing,  we analyzed defect management policies and practices against criteria  from industry best practices. We also compared defect trend data and  metrics over time, and interviewed contractor officials responsible for  testing and defect management. To ensure the reliability of cost and  defect data, we compared data across monthly reports and other agency  data sources, compared formulas from the GAO Cost Guide to the  program’s earned value management approach, and sought  corroboration from agency and contractor officials. To evaluate the  likelihood of a gap in satellite coverage and analyze contingency actions,  we analyzed program documentation to determine the likely length of a  gap, and followed up on our previous effort to compare the GOES-R  contingency plan to best practices in contingency planning identified by  leading organizations. We also interviewed NOAA and GOES-R program  officials regarding program status, defect management, and contingency  planning.
We conducted this performance audit from January 2014 to December  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. See appendix I for a  complete description of our objectives, scope, and methodology.
subsections: section_title: Background
paragraphs: Since the 1970s, geostationary satellites have been used by the United  States to provide meteorological data for weather observation, research,  and forecasting. NOAA’s National Environmental Satellite, Data, and  Information Service is responsible for managing the civilian operational  geostationary satellite system, called GOES. Geostationary satellites can  maintain a constant view of the earth from a high orbit of about 22,300  miles in space.
NOAA operates GOES as a two-satellite system that is primarily focused  on the United States (see fig. 1). These satellites provide timely  environmental data about the earth’s atmosphere, surface, cloud cover,  and the space environment to meteorologists and their audiences. They  also observe the development of hazardous weather, such as hurricanes  and severe thunderstorms, and track their movement and intensity to  reduce or avoid major losses of property and life. The ability of the  satellites to provide broad, continuously updated coverage of atmospheric  conditions over land and oceans is important to NOAA’s weather  forecasting operations.
To provide continuous satellite coverage, NOAA acquires several  satellites at a time as part of a series and launches new satellites every  few years (see table 1). NOAA’s policy is to have two operational  satellites and one backup satellite in orbit at all times.
Three viable GOES satellites—GOES-13, GOES-14, and GOES-15—are  currently in orbit.with GOES-13 covering the eastern United States (GOES-East in  Figure 1) and GOES-15 covering the western United States (GOES- West). GOES-14 is currently in an on-orbit storage mode and is available  as a backup for the other two satellites should they experience any  degradation in service. The GOES-R series is the next generation of  satellites that NOAA is planning.
Both GOES-13 and GOES-15 are operational satellites,  Each of the operational geostationary satellites continuously transmits  raw environmental data to NOAA ground stations. The data are  processed at these ground stations and transmitted back to the satellite  for broadcast to primary weather services and the global research  community in the United States and abroad. Raw and processed data are  also distributed to users via ground stations through other communication  channels, such as dedicated private communication lines and the  Internet. Figure 2 depicts a generic data relay pattern from a  geostationary satellite to the ground stations and commercial terminals.
subsections: section_title: Overview of the GOES-R Program
paragraphs: NOAA established the GOES-R program to develop and launch the next  series of geostationary satellites and to ensure the continuity of  geostationary satellite observations. The GOES-R satellite series is  designed to improve upon the technology of the prior satellite series in  terms of system and instrument improvements. NOAA expects that the  GOES-R series will significantly increase the clarity and precision of the  observed environmental data. In addition, the data generated by the  satellites are to be both developed and transmitted more quickly.
Since its inception, the GOES-R program has undergone several  changes in cost and scope. As originally envisioned, GOES-R was to  encompass four satellites hosting a variety of advanced technology  instruments and providing 81 environmental products. The first two  satellites in the series (called GOES-R and GOES-S) were expected to  launch in September 2012 and April 2014. However, in September 2006,  NOAA decided to reduce the scope and technical complexity of the  GOES-R program because of expectations that total costs, which were  originally estimated to be $6.2 billion, could reach $11.4 billion.  Specifically, NOAA reduced the minimum number of satellites from four to  two, cancelled plans for developing an advanced instrument (which  reduced the number of planned satellite products from 81 to 68), and  divided another instrument into two separate acquisitions. The agency  estimated that the revised program would cost $7 billion and kept the  planned launch dates unchanged.
Subsequently, NOAA made several other important decisions about the  cost and scope of the GOES-R program. In 2007, NOAA established a  new program cost estimate of $7.67 billion and moved the launch dates  for the first two satellites to December 2014 and April 2016. Further, to  mitigate the risk that costs would rise, program officials decided to  remove selected program requirements from the baseline program and  treat them as contract options that could be exercised if funds allowed.  These requirements included the number of products to be distributed,  the time to deliver the remaining products (product latency), and how  often these products would be updated with new satellite data (refresh  rate). For example, program officials eliminated the requirement to  develop and distribute 34 of the 68 envisioned products, including low  cloud and fog, sulfur dioxide detection, and cloud liquid water. Program  officials included the restoration of the requirements for the products,  latency times, and refresh rates as options in the ground system contract  that could be acquired at a later time. Program officials later reduced the  number of products that could be restored as a contract option (called  option 2) from 34 to 31 because they determined that two products were  no longer feasible and two others could be combined into a single  product.
In late 2009, NOAA changed the launch dates for the first two satellites to  October 2015 and February 2017. More recently, NOAA restored two  satellites to the program’s baseline, making GOES-R a four-satellite  program once again. In February 2011, as part of its fiscal year 2012  budget request, NOAA requested funding to begin development for two  additional satellites in the GOES-R series—GOES-T and GOES-U. The  program estimated that the development for all four satellites in the  GOES-R series—GOES-R, GOES-S, GOES-T, and GOES-U—would  cost $10.86 billion through 2036, an increase of $3.19 billion over its 2007  life cycle cost estimate of $7.67 billion for the two-satellite program.
In August 2013, the program announced that it would delay the launch of  the first two satellites in the program, due in part to the effects of  sequestration. Specifically, the launch of the GOES-R satellite was  delayed from October 2015 to the quarter ending March 2016, and the  expected GOES-S satellite launch date was moved from February 2017  to the quarter ending June 2017. See table 2 for an overview of key  changes to the GOES-R program.
section_title: Program Structure
paragraphs: While NOAA is responsible for GOES-R program funding and overall  mission success, it implemented an integrated program management  structure with NASA for the GOES-R program since it relies on NASA’s  acquisition experience and technical expertise. The NOAA-NASA  Program Management Council is the oversight body for the GOES-R  program, and is co-chaired by the NOAA Deputy Undersecretary for  Operations and the NASA Associate Administrator. NOAA also located  the program office at NASA’s Goddard Space Flight Center.
The GOES-R program is divided into flight and ground projects that have  separate areas of responsibility and oversee different sets of contracts.  The flight project, which is led by NASA, includes instruments, spacecraft,  launch services, satellite integration, and on-orbit satellite initialization.  The ground project, which is led by NOAA, is made up of three main  components: the core ground system, an infrastructure of antennas, and  a product access subsystem. In turn, the core ground system comprises  four functional modules supporting operations, product generation,  product distribution, and configuration control. Figure 3 depicts the  integrated program management structure and the organization of the  flight and ground projects within that structure, while table 3 summarizes  the GOES-R instruments and their planned capabilities and table 4  describes key components of the ground project.
section_title: Prior GAO Reports Made Recommendations to Address Program Weaknesses
paragraphs: In recent years, we issued a series of reports aimed at addressing   Key areas of focus included  weaknesses in the GOES-R program.(1) cost, (2) technical challenges and changes in requirements, and  (3) contingency plans.
Addressing cost risks: In June 2012, we reported that the GOES-R  program might not be able to ensure that it had adequate resources to  cover unexpected problems in remaining development. We  recommended the program strengthen its process for planning and  reporting on reserves. More recently, in September 2013, we reported  on weaknesses in the process for reporting reserves to management  and recommended the agency take action to brief senior executives  on a regular basis regarding the status of reserves. The agency  agreed with our recommendations and took steps to address them by  identifying needed reserve levels and providing a more detailed  breakdown of reserve percentage calculations. However, NOAA is not  yet identifying the reserves associated with each satellite in the series.
Technical issues and changes in requirements: We previously  reported on issues related to GOES technical challenges and  requirements. In 2012, we reported that key instruments were  experiencing technical challenges and required additional redesign  efforts. For example, emissions for the Geostationary Lightning   Also, the  Mapper instrument were outside the specified range. ground project was experiencing ongoing technical problems—for  example, the definition of ground system software requirements and  integration of flight instruments. As a result, revisions were made to  the Core Ground System’s baseline development plan and schedule.
More recently, in September 2013, we reported that NOAA made  changes to several of the GOES-R requirements—including  decreasing the accuracy requirement for the hurricane intensity  product and decreasing the timeliness of the lightning detection  product—and that end users were concerned by many of these  changes. We recommended that the program improve communications with users on changes in GOES-R requirements by  assessing impacts, seeking input from users, and disseminating  information on changes. NOAA agreed with this recommendation and  took steps to explore further avenues of user communication such as  customer forums and interagency working groups.
Contingency planning: In February 2013, due to the importance of  environmental satellite data and the potential for a gap in this data, we  added mitigating weather satellite gaps to our biennial High-Risk list.
GAO-12-576.
In that report, we noted that NOAA had established a contingency  plan for a potential gap in the GOES program, but it needed to  demonstrate its progress in coordinating with the user community to  determine their most critical requirements, conducting training and  simulations for contingency operations scenarios, evaluating the  status of viable foreign satellites, and working with the user  community to account for differences in product coverage under  contingency operations scenarios. We also stated that NOAA should  update its contingency plan to provide more details on its contingency  scenarios, associated time frames, and any preventative actions it is  taking to minimize the possibility of a gap.
More recently, in September 2013, we reported that, while NOAA had  established contingency plans for the loss of the GOES satellites,  these plans still did not address user concerns over potential  reductions in capability, and did not identify alternative solutions and  timelines for preventing a delay in the GOES-R launch date.recommended the agency revise the satellite and ground system  contingency plans to address weaknesses, including providing more  information on the potential impact of a satellite failure and  coordinating with key external stakeholders on contingency strategies.  The agency agreed with these recommendations and took steps to  address them by identifying and refining program contingency plans.
section_title: NOAA is Making Progress on the GOES-R Satellite, but Schedule, Cost, and Functionality Challenges Remain
paragraphs: NASA and NOAA are following NASA’s standard space system life cycle  on the GOES-R program. This life cycle includes distinct phases,  including concept and technology development; preliminary design and  technology completion; final design and fabrication; system assembly,  integration and testing, and launch; and operations and sustainment. Key  program reviews are to occur throughout each of the phases, including  preliminary design review, critical design review, and system integration  review. NOAA and NASA jointly conduct key reviews on the flight and  ground segments individually as well as for the program as a whole, and  then make decisions on whether to proceed to the next phase. Figure 4  provides an overview of the life cycle phases, key program reviews, and  associated decision milestones. In addition, the key reviews are  described in table 5.
The GOES-R program has completed important steps in developing its  first satellite. Specifically, the program completed its critical design review  in November 2012, its Mission Operations Review in June 2014, and its  System Integration Review in July 2014. Based on the results of the  System Integration Review, in September 2014, NOAA and NASA  decided to move the program to the next phase, the system assembly,  integration and test, and launch and checkout phase.
To prepare for the recent reviews and milestones, the program completed  numerous important steps on both the flight and ground projects. Key  accomplishments include:  completing testing on individual components including the six flight  instruments, the spacecraft core and system modules, and ground  system components;  releasing key ground system software components on enterprise  infrastructure and mission management in December 2013 and April  2014, respectively, and completing the dry run of another ground  system release that includes all planned products for the GOES-R  satellite;  replacing and successfully demonstrating a new engineering analysis  tool, which will perform trending and offline analysis;  completing installation and testing of antenna dishes at NOAA’s  satellite operations facility, continuing installation and testing at  NOAA’s primary satellite communications site, and beginning training  in use of the antenna system;  completing two key readiness reviews on the product distribution and  access system; and  completing connectivity tests throughout system hardware  components.
Moving forward, the next major program milestones are the flight  operations review and the operational readiness review. In preparation for  these milestones, the program plans to conduct a series of five end-to- end tests.the space and ground segments before the launch of the first satellite.
subsections: section_title: Program Life Cycle Cost Estimate Remains Relatively Steady, but Key Components are Costing More than Expected
paragraphs: NOAA’s estimated life cycle cost for the GOES-R program has held  relatively steady. NOAA’s current life cycle cost estimate for the GOES-R  program is $10.83 billion, which is slightly less than the $10.86 billion  lifecycle cost estimate from August 2013. The $30 million change in the  life cycle cost estimate from last year is the net result of moving selected  management functions outside the program and addressing program  commitments impacted by funding reductions associated with  sequestration in late 2013. However, program data show that individual  components are costing more than expected.
subsections: section_title: Earned Value Management Data Show Key Components are Costing More than Expected
paragraphs: Federal agencies and private industry organizations often implement  earned value management (EVM) as a tool for ensuring work completed  is on track with expected costs and schedules. management tool that, among other things, produces early warning signs  of impending schedule slippages and cost overruns. Key EVM metrics  include cost and schedule variances, which measure the value of work  accomplished in a given period and compares it with the planned value of  work scheduled for that period and with the actual cost of work  accomplished. For example, an increase in cost variance means that the  program spent more than expected to produce the work.
An analysis of EVM data for three key components—the GOES ground  system, the ABI instrument, and the GLM instrument—shows that each  experienced a growing cost variance. Specifically, over the twelve-month  period ending July 2014, the cost variance for the ground system  increased to 8.4 percent of total cumulative budgeted cost and for GLM,  the cost variance increased to 5.1 percent. For a third key component, the  ABI instrument, cost variance increased slightly from 2.4 to 2.6 percent.  Figures 6, 7, and 8 show monthly cost variance data for the three key  components for the year ending July 2014.
GAO-09-3SP.
section_title: Inconsistencies in EVM Data Make Oversight More Difficult
paragraphs: GAO-09-3SP. data for the GLM and ABI instruments we found inconsistencies in the  contractor’s monthly and cumulative reports that made it more difficult for  NOAA to effectively oversee the contractor’s performance. Specifically,  we found inconsistencies between cumulative and monthly budget totals  in contractor performance reports that ranged from hundreds of  thousands to millions of dollars. For instance, the cumulative amount of  budget allocated to complete work for GLM between February 2013 and  March 2013 increased by just under $2 million, while the stated monthly  change was $3.8 million. Also, the cumulative amount of budgeted work  accomplished for ABI between May 2013 and June 2013 increased by  $3.2 million, while the stated monthly change was $5.4 million. Month-to- month discrepancies such as this occurred in each of 6 months for the  GLM instrument and each of 9 months for the ABI instrument in the  period between August 2013 and July 2014.
Program officials stated that these issues were addressed in later  contractor reports, and that program analysts communicate with  contractors and program management regularly to resolve any found  discrepancies. However, more recent monthly reports continue to show  discrepancies. If the instrument’s cost data are unreliable, it is difficult for  managers and program officials to make financial projections and assess  reserve needs and usage.
section_title: The GOES-R Program is Considering Reducing or Deferring Functionality on the Ground System
paragraphs: The GOES-R program is considering eliminating or deferring planned  functionality on its ground system due to issues experienced during  development. Specifically, the GOES-R program is considering deferring  functionality on the ground system in order to provide schedule relief in  the case of further delays. Program and contracting officials recently  revised the composition of the software releases it will be delivering on  the ground system. In doing so, the GOES-R program identified “off- ramps,” or decision points, at which time they could remove or defer a  specific function from pre-launch to post-launch if it is not ready in time for  testing.
As of September 2014, officials identified 50 potential decision points for  deferring functionality. To date, the program has decided to implement  five of the deferrals, including one to remove the ability to play back  information from alternate ABI data sets outside the GOES ground  system, and another to remove a low-level navigation capability. In  addition, program officials decided against deferring 30 of the functions,  leaving 16 deferrals that could be implemented in the future. The off- ramps still under consideration include a reduction in the amount of  verification and validation activities that will be conducted.
section_title: NOAA Has Implemented a Defect Management Process, but Shortfalls Exist and Selected Critical Defects Remain Unresolved
paragraphs: A key element of a successful test phase is appropriately identifying and  handling any defects or anomalies that are discovered during testing. Key  aspects of a sound defect management process include defect  management planning, defect identification and classification, defect  analysis, defect resolution, defect tracking, and defect trending and  reporting. Leading industry and government organizations consider defect  management and resolution to be among the primary goals of testing.These organizations have identified best practices for managing defects  and anomalies that arise during system testing. Table 7 outlines the best  practices of a sound defect management process.
The GOES-R program has sound defect management policies in place  and it, along with its contractors, is actively performing defect  management activities. Specifically, the program has fully satisfied 13 of  the 20 best practices, and partially satisfied the remaining 7 practices. For  example, the program has defect management procedures in place as  part of its overall testing program. Defect management is incorporated in  the program’s mission assurance, configuration management, and  verification/validation functions. In addition, for three key components we  reviewed—ABI, GLM, and the ground system—defects are entered into  automated systems, from which they are analyzed and resolved. The  program also tracks, and regularly reports weekly and monthly, on defect  totals and metrics to NOAA management.
However, there are several areas in which defect management policies  and practices are inconsistent, including in performing and recording  information pertinent to individual defects, and in reporting and tracking  defect information. Table 8 provides an assessment of how the GOES-R  program and key contractors performed on each of the best practices,  and is followed by a more detailed discussion of shortfalls.
Among the shortfalls seen in table 8 are a number of cross-cutting  themes:
Variation among contractors in managing and reporting metrics:  The GOES-R program affords its contractors and subcontractors wide  latitude in making decisions on how to manage, track, and report  defects, which results in variation among program components. For  example, the program established a minimum set of metrics that must  be reported and recorded for each software defect, but has not done  so for hardware defects.
No clear definition of defects: In its guidance, NOAA did not fully  define the terminology of defect management. As a result, the  program and contractors use terms such as defect, anomaly,  nonconformance, incident reports, and trouble reports without  explaining clearly how they are related to each other. Without  understanding these relationships, variations from expected  performance are likely to be treated differently throughout the  program. For example, some issues that occur after formal integration  and testing are complete are not considered as defects, which means  that they are not all reported on the statistics provided to program  management. In another case, an issue on instrument data algorithms  was uncovered during user testing and required rework; however, it  was not counted as a defect. Program officials stated that this was a  documentation issue and noted that they do not track documentation  issues as defects. However, the issue affected more than  documentation. It was addressed by reworking the software  algorithms; thus, it should have been tracked as a software defect.
No clear definition of priority/severity: In its contract requirements  and other guidance documents, the GOES-R program did not  establish specific guidance to its contractors on how to prioritize or  establish the severity of defects. At most, documents list a severity  classification as one of many defect attributes that contractors should  review as necessary. A program official explained that, for hardware  defects, there is no programwide policy for defect metrics, including  priority or severity. As a result, the information tracked, trended, and  reported to management on defect totals also varied by contractor,  and thus often by instrument or component.
Unrealistic testing schedule: Effective defect management requires  a realistic schedule in that it takes time to be able to fully identify,  analyze, prioritize, resolve, and track defects. However, in May 2014,  an internal NOAA report stated that the current program testing  schedule is “unrealistically aggressive.” The GOES-R program has  chosen to compress its remaining testing schedule, which increases  the risk that there will not be sufficient time to address defects before  moving to the next stage of testing. Effects of this type can already be  seen; the program reported that some defects that were open for a  long period of time have been delayed due to the need for all  available resources to be applied in resolving new, current defects.
Limited trend analysis: While the program and NOAA’s mission  assurance team have analyzed contractor-provided trends in the  volume and severity of defects over time for the ground system and  spacecraft, they do not routinely analyze trend data for all  components. For example, on the ABI and GLM instruments, the  program and mission assurance team do not analyze trend data  monthly for hardware defects. Instead, these groups only assess  individual defect reports for this subset of defects. It is important for  the program to assess defect trends over time in order to provide to  management a more complete picture of testing status. Assessing  trends in defect handling can result in better resource allocation to  components of greatest need, and more attention to the effectiveness  of resolving defects.
Although the GOES program has defect management policies in place  and its contractors are actively tracking, analyzing, and reporting on  defects, the discrepancies between contractors’ data could cause issues  in completing the remainder of the program’s integration and testing  period. For example, without consistent defect metrics, it is more difficult  for managers to obtain a complete picture of the status of open, closed,  and high priority defects across the program. Program officials stated that  the program did not contractually specify any best practices, but that it  assumed that contractors with high-level professional certifications would  employ all practices necessary for the development effort.
Unless the program can find a way to unite these disparate approaches to  provide consistency in the methods by which defects are identified,  prioritized, captured, and tracked, it will be more difficult for management  to analyze and understand trends in opening and closing high-priority  defects or to make decisions on how to best resolve the defects.  Moreover, until the program addresses shortfalls in its defect  management processes, it may not have a complete picture of remaining  defects and runs the risk of not having sufficient time to resolve them.
subsections: section_title: Data Show a High Number of Unresolved Defects for Selected Key Components
paragraphs: While effectively and efficiently addressing and resolving defects is an  important part of a sound system testing approach, the GOES-R program  has not efficiently closed defects on selected components. As noted  earlier, the program does not obtain or maintain defect trend data for the  program as a whole; however, data on individual components and  portions of components show that a large number of defects remain open,  including several high-priority defects. Specifically, data for the GOES  ground system shows that 500 defects remained open as of September  2014, including 36 high-priority defects. Defect data for the spacecraft  show that it is taking an increasing amount of time to close hardware- related defects, but that the program is making progress in closing  software defects. Specifically, as of April 2014, 42 software and 332  hardware defects were unresolved.
Defect totals on GOES instruments declined as the program approached  the deadline for them to be completed in order to be integrated with the  spacecraft. Specifically, for the GLM instrument, the total number of both  hardware and software defects declined. Hardware defects declined from  117 in May 2014 to 13 in September 2014, and there were no remaining  software defects by January 2014. For the ABI instrument, the number of  newly opened hardware and software defects each month has declined  over time, with only one unresolved defect remaining in July 2014. Table  9 depicts summary information on defects for selected components at  different points in time. In addition, appendix II provides more information  on defect trends for these components.
Program officials noted that in some cases, due to time concerns, lower- priority defects which have been determined to not have a major effect on  performance are not closed until later in the testing process. Also,  program officials have stated that they are having difficulty in closing  defect-related incident reports due to insufficient manpower. Until the  program reduces the number of longstanding unresolved defects, it faces  an increased risk of further delays to the GOES-R launch date should an  open defect affect future performance.
section_title: Facing a Gap in Backup Satellite Coverage, the GOES-R Program Has Improved Contingency Plans Though Shortfalls Remain
paragraphs: GOES satellite data are considered a mission-essential function because  of their criticality to weather observations and forecasts. These  forecasts—such as those for severe storms, hurricanes, and tornadoes— can have a substantial impact on our nation’s people, infrastructure, and  economy. Because of the importance of GOES satellite data, NOAA’s  policy is to have two operational satellites and one backup satellite in orbit  at all times. This policy proved useful in December 2008 and again in  September 2012, when the agency experienced problems with one of its  operational satellites, but was able to move its backup satellite into place  until the problems had been resolved.
However, NOAA is facing a period of up to 17 months when it will not  have a backup satellite in orbit. Specifically, in April 2015, NOAA expects  to retire one of its operational satellites (GOES-13) and to move its  backup satellite (GOES-14) into operation. Thus, the agency will have  only two operational satellites in orbit—and no backup satellite—until  GOES-R is launched and completes an estimated 6-month post-launch  test period. If GOES-R is launched in March 2016, the earliest it could be  available for operational use would be September 2016. Figure 9 shows  the potential gap in backup coverage, based on the launch and  decommission dates of GOES satellites.
During the time in which no backup satellite would be available, there is a  greater risk that NOAA would need to either rely on older satellites that  are beyond their expected operational lives and may not be fully  functional, request foreign satellite coverage, or to operate with only a  single operational satellite. Agency officials stated that the risk of a gap  may be reduced, because NOAA satellites have historically remained in  operation longer than their expected life. While many satellites outlive  their expected lifespans, the current GOES satellites are operating with  reduced functionality, and one has experienced two major outages.  Without a full complement of operational GOES satellites, the nation’s  ability to maintain the continuity of data required for effective weather  forecasting could be compromised. This, in turn, could put the public,  property, and the economy at risk.
Any delay to the GOES-R launch date would extend the time without a  backup to more than 17 months. As discussed earlier in this report,  further delays to the committed launch date of the first GOES-R satellite  are possible due to continued technical issues encountered during testing  and integration.
subsections: section_title: GOES-R Satellite Contingency Plan Shows Improvement, but Continues to Lack Details in Key Areas
paragraphs: Government and industry best practices call for the development of  contingency plans to maintain an organization’s essential functions—such  as GOES satellite data—in the case of an adverse event. In September  2013, we reported on weaknesses in the contingency plans for NOAA’s  satellites. At that time, we compared NOAA’s plans to 17 best practices  associated with three main areas: identifying failure scenarios and  impacts, developing contingency plans, and validating and implementing  contingency plans. We reported that while NOAA identified failure  scenarios, recovery priorities, and minimum levels of acceptable  performance, the satellite contingency plan contained areas that fell short  of best practices, such as working with the user community to account for  potential reductions in capability under contingency operations.  Furthermore, the agency did not identify alternative solutions or timelines  for preventing a GOES-R launch delay.
In February 2014, NOAA released a new satellite contingency plan in  response to these recommendations. This plan improved in comparison  to many, but not all, of the best practices. Specifically, the plan improved  in 6 areas and stayed the same in 4 areas. Table 10 compares our  assessment of the current satellite contingency plan with our September  2013 analysis for all best practices that were not fully met.
Program officials stated that it is not feasible to include strategies to  prevent delays in launch of the first GOES-R satellite in the contingency  plan, because such strategies are not static. They explained that options  for preventing a delay vary greatly over time based on issues that occur  during the satellite’s development. They further stated that the program’s  monthly presentations to NOAA management provide a summary of  current threats to the launch date and strategies to mitigate those threats.  For example, NOAA is considering whether or not to remove or delay  selected ground system functions to post-launch, and the program  provides monthly updates on this issue.
While actively managing the program to avoid a delay is critical, it is also  important that NOAA management and the GOES-R program consider  and document feasible alternatives for avoiding or limiting such a launch  delay. This will allow stakeholders throughout NOAA to be aware of,  respond to, and plan for the potential implementation of each alternative,  not only the small number of alternatives the program is actively  considering in any given month. Until NOAA addresses the remaining  shortfalls in its GOES-R gap mitigation plan, the agency cannot be  assured that it is exploring all alternatives or that are able to effectively  prepare to receive GOES information in the event of a failure.
section_title: Conclusions
paragraphs: After spending 10 years and just over $5 billion, the GOES-R program is  nearing the launch of its first satellite. However, it continues to face  challenges in maintaining its schedule and controlling its costs. The  program continues to experience delays in remaining major milestones,  which could result in further delays to the launch date. Costs are  increasing faster than expected for key program components and  contractor data is often inconsistent from month to month. Until the  agency ensures its contractor cost data are consistent, it will be more  difficult for managers and program officials to make financial projections  and assess reserve needs and usage.
As the GOES-R program progresses through its testing and integration  phase, it is essential that it is able to appropriately handle defects that  arise during testing. While NOAA and its contractors have implemented a  defect management process which is successful in many areas, there are  shortfalls in how the program defines defects, monitors trends, and  reports on defects and defect metrics. In particular, NOAA has not  established a standard set of metric information for all defects, including  the dates on which defects were identified and resolved, or an indication  of a defect’s severity. Also, NOAA did not clearly define the type of issue  that constitutes a defect. Furthermore, multiple defects remain open on  some program components, including several that have remained open  for more than six months. Until the program addresses these shortfalls  and reduces the number of open defects, it may not have a complete  picture of remaining issues and faces an increased risk of further delays  to the GOES-R launch date.
NOAA could experience a gap in satellite data coverage if GOES-R is  delayed further and one of the two remaining operational satellites  experiences a problem. NOAA has made improvements to its satellite  contingency plan, but the plan still does not sufficiently address mitigation  options for a launch delay, potential impacts, or minimum performance  levels. Until such information is available, it will be difficult to integrate  mitigation efforts, or to coordinate with users in the event of a failure.
section_title: Recommendations for Executive Action
paragraphs: To address risks in the GOES-R program development and to help  ensure that the satellite is launched on time, we are making the following  four recommendations to the Secretary of Commerce. Specifically, we  recommend that the Secretary of Commerce direct the NOAA  Administrator to:  investigate and address inconsistencies totaling hundreds of  thousands of dollars in monthly earned value data reporting for the  GLM and ABI instruments; address shortfalls in defect management identified in this report,  including the lack of clear guidance on defect definitions, what defect  metrics should be collected and reported, and how to establish a  defect’s priority or severity; and  reduce the number of unresolved defects on the GOES ground  system and spacecraft.
In addition, because NOAA has not fully implemented our prior  recommendation to improve its satellite gap mitigation plan, we  recommend that the Secretary of Commerce direct the NOAA  Administrator to: add information to the GOES satellite contingency plan on steps  planned or underway to mitigate potential launch delays, the potential  impact of failure scenarios in the plan, and the minimum performance  levels expected under such scenarios.
section_title: Agency Comments and Our Evaluation
paragraphs: We sought comments on a draft of our report from the Department of  Commerce and NASA. We received written comments from the Deputy  Secretary of Commerce transmitting NOAA’s comments. NOAA  concurred with all four of our recommendations and identified steps that it  plans to take to implement them. It also provided technical comments,  which we have incorporated into our report, as appropriate. NOAA’s  comments are reprinted in appendix III.
On November 14, 2014, an audit liaison for NASA provided an e-mail  stating that the agency would provide any input it might have to NOAA for  inclusion in NOAA’s comments.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to interested congressional  committees, the Secretary of Commerce, the Administrator of NASA, the  Director of the Office of Management and Budget, and other interested  parties. The report also will be available at no charge on the GAO website  at http://www.gao.gov.
If you or your staff have any questions on the matters discussed in this  report, please contact me at (202) 512-9286 or at pownerd@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) assess progress on the GOES-R program with  respect to planned schedule, cost, and functionality; (2) assess efforts to  identify and address issues discovered during integration and testing; and  (3) evaluate the likelihood of a gap in satellite coverage and analyze the  adequacy of contingency actions in place to prevent or mitigate such a  gap.
To assess NOAA’s progress on the GOES-R program with respect to  planned schedule, cost, and functionality, and to identify risks that could  lead to further schedule delays, we analyzed data from monthly program  management meetings. We evaluated progress made in completing key  program components and major program reviews, and compared planned  and actual completion dates for key program milestones over the last two  to three years to determine the degree to which these dates have  changed. To ensure that the program’s schedule data were consistent  and reliable, we compared milestone data over several months and  contacted agency officials to corroborate events that occurred over the  course of our engagement. We analyzed earned value management  (EVM) data to compare levels of cost variance over time for key program  components, and calculated earned value management metrics using  program cost performance reports. To ensure that the program’s cost  data were reliable, we compared formulas from the GAO Cost Guide to  the program’s EVM approach. We also compared EVM data across a  series of monthly program cost performance reports. In doing so, we  found inconsistencies in the monthly EVM data for selected components,  and reported on those inconsistencies in this report. However, the data  were sufficient for our purpose of assessing overruns because these  inconsistencies were small in comparison to cost variances. We assessed  recently enacted and potential changes in functionality for key program  components. We also interviewed program officials regarding changes in  schedule milestones, cost performance and reserve funding, and  functionality.
To assess efforts to identify and address issues discovered during  integration and testing, we identified best practices in defect management  from leading industry and government organizations. We compared  NOAA policy documents and defect management artifacts by the GOES  program, its contractors, and an independent NOAA mission assurance  group to the best practices. We selected three components, which are  critical to the program’s mission requirements, and evaluated recent test  results and defects. Specifically, we identified two recent tests for each  component and analyzed artifacts associated with defects identified  during those tests. We compared the defects against each best practice  to determine if NOAA and its contractors fully implemented, partially  implemented, or did not implement the best practices. The agency and  contractor had to meet all aspects of the best practice to achieve a fully  implemented score, some aspects to achieve a partially implemented  score, and no aspects to achieve a not implemented score. To identify the  program’s recent performance in closing and managing defects, we  analyzed basic defect trend information—such as number of defects  opened and closed, and defect severity—for each of several program  components. We developed charts to demonstrate defect trends. To  ensure the data were reliable, we compared defect data from individual  defect reports to agency trend charts, and sought corroboration from  agency and contractor officials. We also interviewed agency and  contractor officials to discuss their defect management processes and  practices, and to confirm information we found while analyzing individual  defect reports and trend charts.
To evaluate the likelihood of a gap in satellite coverage, we reviewed  monthly program management presentations and other review board  documentation. To analyze the adequacy of contingency actions in place  to prevent or mitigate such a gap, we compared NOAA’s latest satellite  contingency plan to best practice criteria from industry and government.  We focused specifically on the ten areas we identified as weak in our  prior report. For each of the ten areas, we rated NOAA’s contingency  plan as having partially or fully implemented the best practice criteria. In  addition, we interviewed agency officials regarding the likely amount of a  gap in on-orbit backup coverage of GOES satellites, the potential for a  gap in operational coverage, and efforts to improve the GOES  contingency plan.
We conducted this performance audit from January 2014 to December  2014, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Defect Trends for Key GOES-R Components
paragraphs: As noted earlier, the GOES-R program does not obtain or maintain defect  trend data for the program as a whole; however, data on individual  components and portions of components show that a large number of  defects remain open, including several high-priority defects. The following  sections provide more details on recent data trends for the GOES-R  ground system and spacecraft as well as the ABI and GLM instruments. It  is important to note that the sections do not provide consistent information  among components because the program does not require consistent  metrics and the contractors document different data.
subsections: section_title: Ground System Has Many Unresolved Defects
paragraphs: As of the end of September 2014, the ground system had 500 open  defects, an increase from 342 open defects at the end of September  2013. Also, as of the end of September 2014, 96 defects that were  identified prior to January 2014 remained open, and 36 high-priority  defects (those rated as critical or moderate) remained open.
Program and contractor officials provided rationale and insight into the  mitigating circumstances surrounding these defects. Program officials  stated that the number of open defects during the integration and test  phase is as expected for a ground system of a magnitude similar to  GOES. Contractor officials reported that none of the 96 longstanding  defects were in either of the two highest severity categories. They also  noted that most of the open high-priority defects had been opened during  testing events conducted over the previous 3 months, which is a  reasonable length of time to close a defect.
Furthermore, according to contractor officials, many of the longstanding  open defects were in categories such as documentation-related defects  which are considered less severe. Officials also stated that defects in  these categories are often kept open for specific reasons, such as to gain  cost efficiencies or to wait for an already-planned test event rather than  creating a new test event. However, as of September 2013, 193 of the  342 defects were not related to documentation and this number rose to  416 of the 500 open defects in September 2014. Figure 10 shows the  increase in open defects in the period from September 2013 to August  2014 for the GOES-R ground system.
As allowed by NOAA’s guidance, contractors on the spacecraft track  defects differently than the ground system contractors do. Thus, it is not  possible to compare opened and closed defects over time as depicted  above for the ground system. Instead, spacecraft contractors track  hardware and software defects independently. They also track when new  defects are identified and how long they stay open.
For spacecraft hardware, the defect data show that the average time it  took to resolve defects increased over time. Specifically, the average age  for hardware defects increased from 99 days in May 2013 to a high of 167  days in April 2014, at which point 58 percent of all open defects had been  open for at least 90 days. Program officials stated that they began  continuous 24-hour a day, 7-day a week testing in October 2014, which  means that they should be able to make progress in closing the remaining  defects.
While the backlog of hardware-related defects on the spacecraft remains  high, NOAA has been effective in recent months in greatly reducing the  number of software-related defects on the spacecraft. The total number of  open software-related defects increased and remained high through  February 2014, but then declined significantly in March and April 2014.
section_title: Defect Trends Show Decline in Defects for GLM Instrument
paragraphs: Defect data for the GLM instrument showed a decline in the total number  of defects for both hardware and software components. While there were  117 unresolved hardware defects in May 2014, only 13 hardware defects  remained unresolved as of September 2014. Trend data also showed a  decline in the number of unresolved software defects. Specifically, the  total number of software-related defects remaining open never increased  above 7 over the period from March 2013 to January 2014. For half the  year, there was no more than one open defect.
section_title: ABI Hardware and Software Defects Were Closed in Recent Months
paragraphs: Metrics for the ABI instrument show that all hardware defects and all but  one software defect have been closed, likely because the first unit of the  instrument was completed. The number of hardware defects occurring  each month declined to near zero during the period February 2013 to  April 2014. Less than ten ABI software defects were open at any point  from September 2012 onward, and only 11 defects were newly opened  during that time. Figure 11 shows the number of opened and closed ABI  hardware defects by month, and figure 12 shows opened and closed  defects for ABI software.
section_title: Appendix III: Comments from the Department of Commerce
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, individuals making contributions  to this report included Colleen Phillips (assistant director), Alexander  Anderegg, Christopher Businsky, Shaun Byrnes, James MacAulay, and  Karl Seifert.