Geostationary Weather Satellites: Launch Date Nears, but Remaining Schedule Risks Need to be Addressed
Released: 2015-01-15T00:00:00

Letter

Background

Since the 1970s, geostationary satellites have been used by the United States to provide meteorological data for weather observation, research, and forecasting. NOAA’s National Environmental Satellite, Data, and Information Service is responsible for managing the civilian operational geostationary satellite system, called GOES. Geostationary satellites can maintain a constant view of the earth from a high orbit of about 22,300 miles in space.

NOAA operates GOES as a two-satellite system that is primarily focused on the United States (see fig. 1). These satellites provide timely environmental data about the earth’s atmosphere, surface, cloud cover, and the space environment to meteorologists and their audiences. They also observe the development of hazardous weather, such as hurricanes and severe thunderstorms, and track their movement and intensity to reduce or avoid major losses of property and life. The ability of the satellites to provide broad, continuously updated coverage of atmospheric conditions over land and oceans is important to NOAA’s weather forecasting operations.

To provide continuous satellite coverage, NOAA acquires several satellites at a time as part of a series and launches new satellites every few years (see table 1). NOAA’s policy is to have two operational satellites and one backup satellite in orbit at all times.

Three viable GOES satellites—GOES-13, GOES-14, and GOES-15—are currently in orbit.with GOES-13 covering the eastern United States (GOES-East in Figure 1) and GOES-15 covering the western United States (GOES- West). GOES-14 is currently in an on-orbit storage mode and is available as a backup for the other two satellites should they experience any degradation in service. The GOES-R series is the next generation of satellites that NOAA is planning.

Both GOES-13 and GOES-15 are operational satellites, Each of the operational geostationary satellites continuously transmits raw environmental data to NOAA ground stations. The data are processed at these ground stations and transmitted back to the satellite for broadcast to primary weather services and the global research community in the United States and abroad. Raw and processed data are also distributed to users via ground stations through other communication channels, such as dedicated private communication lines and the Internet. Figure 2 depicts a generic data relay pattern from a geostationary satellite to the ground stations and commercial terminals.

Overview of the GOES-R Program

NOAA established the GOES-R program to develop and launch the next series of geostationary satellites and to ensure the continuity of geostationary satellite observations. The GOES-R satellite series is designed to improve upon the technology of the prior satellite series in terms of system and instrument improvements. NOAA expects that the GOES-R series will significantly increase the clarity and precision of the observed environmental data. In addition, the data generated by the satellites are to be both developed and transmitted more quickly.

Since its inception, the GOES-R program has undergone several changes in cost and scope. As originally envisioned, GOES-R was to encompass four satellites hosting a variety of advanced technology instruments and providing 81 environmental products. The first two satellites in the series (called GOES-R and GOES-S) were expected to launch in September 2012 and April 2014. However, in September 2006, NOAA decided to reduce the scope and technical complexity of the GOES-R program because of expectations that total costs, which were originally estimated to be $6.2 billion, could reach $11.4 billion. Specifically, NOAA reduced the minimum number of satellites from four to two, cancelled plans for developing an advanced instrument (which reduced the number of planned satellite products from 81 to 68), and divided another instrument into two separate acquisitions. The agency estimated that the revised program would cost $7 billion and kept the planned launch dates unchanged.

Subsequently, NOAA made several other important decisions about the cost and scope of the GOES-R program. In 2007, NOAA established a new program cost estimate of $7.67 billion and moved the launch dates for the first two satellites to December 2014 and April 2016. Further, to mitigate the risk that costs would rise, program officials decided to remove selected program requirements from the baseline program and treat them as contract options that could be exercised if funds allowed. These requirements included the number of products to be distributed, the time to deliver the remaining products (product latency), and how often these products would be updated with new satellite data (refresh rate). For example, program officials eliminated the requirement to develop and distribute 34 of the 68 envisioned products, including low cloud and fog, sulfur dioxide detection, and cloud liquid water. Program officials included the restoration of the requirements for the products, latency times, and refresh rates as options in the ground system contract that could be acquired at a later time. Program officials later reduced the number of products that could be restored as a contract option (called option 2) from 34 to 31 because they determined that two products were no longer feasible and two others could be combined into a single product.

In late 2009, NOAA changed the launch dates for the first two satellites to October 2015 and February 2017. More recently, NOAA restored two satellites to the program’s baseline, making GOES-R a four-satellite program once again. In February 2011, as part of its fiscal year 2012 budget request, NOAA requested funding to begin development for two additional satellites in the GOES-R series—GOES-T and GOES-U. The program estimated that the development for all four satellites in the GOES-R series—GOES-R, GOES-S, GOES-T, and GOES-U—would cost $10.86 billion through 2036, an increase of $3.19 billion over its 2007 life cycle cost estimate of $7.67 billion for the two-satellite program.

In August 2013, the program announced that it would delay the launch of the first two satellites in the program, due in part to the effects of sequestration. Specifically, the launch of the GOES-R satellite was delayed from October 2015 to the quarter ending March 2016, and the expected GOES-S satellite launch date was moved from February 2017 to the quarter ending June 2017. See table 2 for an overview of key changes to the GOES-R program.

Program Structure

While NOAA is responsible for GOES-R program funding and overall mission success, it implemented an integrated program management structure with NASA for the GOES-R program since it relies on NASA’s acquisition experience and technical expertise. The NOAA-NASA Program Management Council is the oversight body for the GOES-R program, and is co-chaired by the NOAA Deputy Undersecretary for Operations and the NASA Associate Administrator. NOAA also located the program office at NASA’s Goddard Space Flight Center.

The GOES-R program is divided into flight and ground projects that have separate areas of responsibility and oversee different sets of contracts. The flight project, which is led by NASA, includes instruments, spacecraft, launch services, satellite integration, and on-orbit satellite initialization. The ground project, which is led by NOAA, is made up of three main components: the core ground system, an infrastructure of antennas, and a product access subsystem. In turn, the core ground system comprises four functional modules supporting operations, product generation, product distribution, and configuration control. Figure 3 depicts the integrated program management structure and the organization of the flight and ground projects within that structure, while table 3 summarizes the GOES-R instruments and their planned capabilities and table 4 describes key components of the ground project.

Prior GAO Reports Made Recommendations to Address Program Weaknesses

In recent years, we issued a series of reports aimed at addressing Key areas of focus included weaknesses in the GOES-R program.(1) cost, (2) technical challenges and changes in requirements, and (3) contingency plans.

Addressing cost risks: In June 2012, we reported that the GOES-R program might not be able to ensure that it had adequate resources to cover unexpected problems in remaining development. We recommended the program strengthen its process for planning and reporting on reserves. More recently, in September 2013, we reported on weaknesses in the process for reporting reserves to management and recommended the agency take action to brief senior executives on a regular basis regarding the status of reserves. The agency agreed with our recommendations and took steps to address them by identifying needed reserve levels and providing a more detailed breakdown of reserve percentage calculations. However, NOAA is not yet identifying the reserves associated with each satellite in the series.

Technical issues and changes in requirements: We previously reported on issues related to GOES technical challenges and requirements. In 2012, we reported that key instruments were experiencing technical challenges and required additional redesign efforts. For example, emissions for the Geostationary Lightning Also, the Mapper instrument were outside the specified range. ground project was experiencing ongoing technical problems—for example, the definition of ground system software requirements and integration of flight instruments. As a result, revisions were made to the Core Ground System’s baseline development plan and schedule.

More recently, in September 2013, we reported that NOAA made changes to several of the GOES-R requirements—including decreasing the accuracy requirement for the hurricane intensity product and decreasing the timeliness of the lightning detection product—and that end users were concerned by many of these changes. We recommended that the program improve communications with users on changes in GOES-R requirements by assessing impacts, seeking input from users, and disseminating information on changes. NOAA agreed with this recommendation and took steps to explore further avenues of user communication such as customer forums and interagency working groups.

Contingency planning: In February 2013, due to the importance of environmental satellite data and the potential for a gap in this data, we added mitigating weather satellite gaps to our biennial High-Risk list.

GAO-12-576.

In that report, we noted that NOAA had established a contingency plan for a potential gap in the GOES program, but it needed to demonstrate its progress in coordinating with the user community to determine their most critical requirements, conducting training and simulations for contingency operations scenarios, evaluating the status of viable foreign satellites, and working with the user community to account for differences in product coverage under contingency operations scenarios. We also stated that NOAA should update its contingency plan to provide more details on its contingency scenarios, associated time frames, and any preventative actions it is taking to minimize the possibility of a gap.

More recently, in September 2013, we reported that, while NOAA had established contingency plans for the loss of the GOES satellites, these plans still did not address user concerns over potential reductions in capability, and did not identify alternative solutions and timelines for preventing a delay in the GOES-R launch date.recommended the agency revise the satellite and ground system contingency plans to address weaknesses, including providing more information on the potential impact of a satellite failure and coordinating with key external stakeholders on contingency strategies. The agency agreed with these recommendations and took steps to address them by identifying and refining program contingency plans.

NOAA is Making Progress on the GOES-R Satellite, but Schedule, Cost, and Functionality Challenges Remain

NASA and NOAA are following NASA’s standard space system life cycle on the GOES-R program. This life cycle includes distinct phases, including concept and technology development; preliminary design and technology completion; final design and fabrication; system assembly, integration and testing, and launch; and operations and sustainment. Key program reviews are to occur throughout each of the phases, including preliminary design review, critical design review, and system integration review. NOAA and NASA jointly conduct key reviews on the flight and ground segments individually as well as for the program as a whole, and then make decisions on whether to proceed to the next phase. Figure 4 provides an overview of the life cycle phases, key program reviews, and associated decision milestones. In addition, the key reviews are described in table 5.

The GOES-R program has completed important steps in developing its first satellite. Specifically, the program completed its critical design review in November 2012, its Mission Operations Review in June 2014, and its System Integration Review in July 2014. Based on the results of the System Integration Review, in September 2014, NOAA and NASA decided to move the program to the next phase, the system assembly, integration and test, and launch and checkout phase.

To prepare for the recent reviews and milestones, the program completed numerous important steps on both the flight and ground projects. Key accomplishments include: completing testing on individual components including the six flight instruments, the spacecraft core and system modules, and ground system components; releasing key ground system software components on enterprise infrastructure and mission management in December 2013 and April 2014, respectively, and completing the dry run of another ground system release that includes all planned products for the GOES-R satellite; replacing and successfully demonstrating a new engineering analysis tool, which will perform trending and offline analysis; completing installation and testing of antenna dishes at NOAA’s satellite operations facility, continuing installation and testing at NOAA’s primary satellite communications site, and beginning training in use of the antenna system; completing two key readiness reviews on the product distribution and access system; and completing connectivity tests throughout system hardware components.

Moving forward, the next major program milestones are the flight operations review and the operational readiness review. In preparation for these milestones, the program plans to conduct a series of five end-to- end tests.the space and ground segments before the launch of the first satellite.

Program Life Cycle Cost Estimate Remains Relatively Steady, but Key Components are Costing More than Expected

NOAA’s estimated life cycle cost for the GOES-R program has held relatively steady. NOAA’s current life cycle cost estimate for the GOES-R program is $10.83 billion, which is slightly less than the $10.86 billion lifecycle cost estimate from August 2013. The $30 million change in the life cycle cost estimate from last year is the net result of moving selected management functions outside the program and addressing program commitments impacted by funding reductions associated with sequestration in late 2013. However, program data show that individual components are costing more than expected.

Earned Value Management Data Show Key Components are Costing More than Expected

Federal agencies and private industry organizations often implement earned value management (EVM) as a tool for ensuring work completed is on track with expected costs and schedules. management tool that, among other things, produces early warning signs of impending schedule slippages and cost overruns. Key EVM metrics include cost and schedule variances, which measure the value of work accomplished in a given period and compares it with the planned value of work scheduled for that period and with the actual cost of work accomplished. For example, an increase in cost variance means that the program spent more than expected to produce the work.

An analysis of EVM data for three key components—the GOES ground system, the ABI instrument, and the GLM instrument—shows that each experienced a growing cost variance. Specifically, over the twelve-month period ending July 2014, the cost variance for the ground system increased to 8.4 percent of total cumulative budgeted cost and for GLM, the cost variance increased to 5.1 percent. For a third key component, the ABI instrument, cost variance increased slightly from 2.4 to 2.6 percent. Figures 6, 7, and 8 show monthly cost variance data for the three key components for the year ending July 2014.

GAO-09-3SP.

Inconsistencies in EVM Data Make Oversight More Difficult

GAO-09-3SP. data for the GLM and ABI instruments we found inconsistencies in the contractor’s monthly and cumulative reports that made it more difficult for NOAA to effectively oversee the contractor’s performance. Specifically, we found inconsistencies between cumulative and monthly budget totals in contractor performance reports that ranged from hundreds of thousands to millions of dollars. For instance, the cumulative amount of budget allocated to complete work for GLM between February 2013 and March 2013 increased by just under $2 million, while the stated monthly change was $3.8 million. Also, the cumulative amount of budgeted work accomplished for ABI between May 2013 and June 2013 increased by $3.2 million, while the stated monthly change was $5.4 million. Month-to- month discrepancies such as this occurred in each of 6 months for the GLM instrument and each of 9 months for the ABI instrument in the period between August 2013 and July 2014.

Program officials stated that these issues were addressed in later contractor reports, and that program analysts communicate with contractors and program management regularly to resolve any found discrepancies. However, more recent monthly reports continue to show discrepancies. If the instrument’s cost data are unreliable, it is difficult for managers and program officials to make financial projections and assess reserve needs and usage.

The GOES-R Program is Considering Reducing or Deferring Functionality on the Ground System

The GOES-R program is considering eliminating or deferring planned functionality on its ground system due to issues experienced during development. Specifically, the GOES-R program is considering deferring functionality on the ground system in order to provide schedule relief in the case of further delays. Program and contracting officials recently revised the composition of the software releases it will be delivering on the ground system. In doing so, the GOES-R program identified “off- ramps,” or decision points, at which time they could remove or defer a specific function from pre-launch to post-launch if it is not ready in time for testing.

As of September 2014, officials identified 50 potential decision points for deferring functionality. To date, the program has decided to implement five of the deferrals, including one to remove the ability to play back information from alternate ABI data sets outside the GOES ground system, and another to remove a low-level navigation capability. In addition, program officials decided against deferring 30 of the functions, leaving 16 deferrals that could be implemented in the future. The off- ramps still under consideration include a reduction in the amount of verification and validation activities that will be conducted.

NOAA Has Implemented a Defect Management Process, but Shortfalls Exist and Selected Critical Defects Remain Unresolved

A key element of a successful test phase is appropriately identifying and handling any defects or anomalies that are discovered during testing. Key aspects of a sound defect management process include defect management planning, defect identification and classification, defect analysis, defect resolution, defect tracking, and defect trending and reporting. Leading industry and government organizations consider defect management and resolution to be among the primary goals of testing.These organizations have identified best practices for managing defects and anomalies that arise during system testing. Table 7 outlines the best practices of a sound defect management process.

The GOES-R program has sound defect management policies in place and it, along with its contractors, is actively performing defect management activities. Specifically, the program has fully satisfied 13 of the 20 best practices, and partially satisfied the remaining 7 practices. For example, the program has defect management procedures in place as part of its overall testing program. Defect management is incorporated in the program’s mission assurance, configuration management, and verification/validation functions. In addition, for three key components we reviewed—ABI, GLM, and the ground system—defects are entered into automated systems, from which they are analyzed and resolved. The program also tracks, and regularly reports weekly and monthly, on defect totals and metrics to NOAA management.

However, there are several areas in which defect management policies and practices are inconsistent, including in performing and recording information pertinent to individual defects, and in reporting and tracking defect information. Table 8 provides an assessment of how the GOES-R program and key contractors performed on each of the best practices, and is followed by a more detailed discussion of shortfalls.

Among the shortfalls seen in table 8 are a number of cross-cutting themes:

Variation among contractors in managing and reporting metrics: The GOES-R program affords its contractors and subcontractors wide latitude in making decisions on how to manage, track, and report defects, which results in variation among program components. For example, the program established a minimum set of metrics that must be reported and recorded for each software defect, but has not done so for hardware defects.

No clear definition of defects: In its guidance, NOAA did not fully define the terminology of defect management. As a result, the program and contractors use terms such as defect, anomaly, nonconformance, incident reports, and trouble reports without explaining clearly how they are related to each other. Without understanding these relationships, variations from expected performance are likely to be treated differently throughout the program. For example, some issues that occur after formal integration and testing are complete are not considered as defects, which means that they are not all reported on the statistics provided to program management. In another case, an issue on instrument data algorithms was uncovered during user testing and required rework; however, it was not counted as a defect. Program officials stated that this was a documentation issue and noted that they do not track documentation issues as defects. However, the issue affected more than documentation. It was addressed by reworking the software algorithms; thus, it should have been tracked as a software defect.

No clear definition of priority/severity: In its contract requirements and other guidance documents, the GOES-R program did not establish specific guidance to its contractors on how to prioritize or establish the severity of defects. At most, documents list a severity classification as one of many defect attributes that contractors should review as necessary. A program official explained that, for hardware defects, there is no programwide policy for defect metrics, including priority or severity. As a result, the information tracked, trended, and reported to management on defect totals also varied by contractor, and thus often by instrument or component.

Unrealistic testing schedule: Effective defect management requires a realistic schedule in that it takes time to be able to fully identify, analyze, prioritize, resolve, and track defects. However, in May 2014, an internal NOAA report stated that the current program testing schedule is “unrealistically aggressive.” The GOES-R program has chosen to compress its remaining testing schedule, which increases the risk that there will not be sufficient time to address defects before moving to the next stage of testing. Effects of this type can already be seen; the program reported that some defects that were open for a long period of time have been delayed due to the need for all available resources to be applied in resolving new, current defects.

Limited trend analysis: While the program and NOAA’s mission assurance team have analyzed contractor-provided trends in the volume and severity of defects over time for the ground system and spacecraft, they do not routinely analyze trend data for all components. For example, on the ABI and GLM instruments, the program and mission assurance team do not analyze trend data monthly for hardware defects. Instead, these groups only assess individual defect reports for this subset of defects. It is important for the program to assess defect trends over time in order to provide to management a more complete picture of testing status. Assessing trends in defect handling can result in better resource allocation to components of greatest need, and more attention to the effectiveness of resolving defects.

Although the GOES program has defect management policies in place and its contractors are actively tracking, analyzing, and reporting on defects, the discrepancies between contractors’ data could cause issues in completing the remainder of the program’s integration and testing period. For example, without consistent defect metrics, it is more difficult for managers to obtain a complete picture of the status of open, closed, and high priority defects across the program. Program officials stated that the program did not contractually specify any best practices, but that it assumed that contractors with high-level professional certifications would employ all practices necessary for the development effort.

Unless the program can find a way to unite these disparate approaches to provide consistency in the methods by which defects are identified, prioritized, captured, and tracked, it will be more difficult for management to analyze and understand trends in opening and closing high-priority defects or to make decisions on how to best resolve the defects. Moreover, until the program addresses shortfalls in its defect management processes, it may not have a complete picture of remaining defects and runs the risk of not having sufficient time to resolve them.

Data Show a High Number of Unresolved Defects for Selected Key Components

While effectively and efficiently addressing and resolving defects is an important part of a sound system testing approach, the GOES-R program has not efficiently closed defects on selected components. As noted earlier, the program does not obtain or maintain defect trend data for the program as a whole; however, data on individual components and portions of components show that a large number of defects remain open, including several high-priority defects. Specifically, data for the GOES ground system shows that 500 defects remained open as of September 2014, including 36 high-priority defects. Defect data for the spacecraft show that it is taking an increasing amount of time to close hardware- related defects, but that the program is making progress in closing software defects. Specifically, as of April 2014, 42 software and 332 hardware defects were unresolved.

Defect totals on GOES instruments declined as the program approached the deadline for them to be completed in order to be integrated with the spacecraft. Specifically, for the GLM instrument, the total number of both hardware and software defects declined. Hardware defects declined from 117 in May 2014 to 13 in September 2014, and there were no remaining software defects by January 2014. For the ABI instrument, the number of newly opened hardware and software defects each month has declined over time, with only one unresolved defect remaining in July 2014. Table 9 depicts summary information on defects for selected components at different points in time. In addition, appendix II provides more information on defect trends for these components.

Program officials noted that in some cases, due to time concerns, lower- priority defects which have been determined to not have a major effect on performance are not closed until later in the testing process. Also, program officials have stated that they are having difficulty in closing defect-related incident reports due to insufficient manpower. Until the program reduces the number of longstanding unresolved defects, it faces an increased risk of further delays to the GOES-R launch date should an open defect affect future performance.

Facing a Gap in Backup Satellite Coverage, the GOES-R Program Has Improved Contingency Plans Though Shortfalls Remain

GOES satellite data are considered a mission-essential function because of their criticality to weather observations and forecasts. These forecasts—such as those for severe storms, hurricanes, and tornadoes— can have a substantial impact on our nation’s people, infrastructure, and economy. Because of the importance of GOES satellite data, NOAA’s policy is to have two operational satellites and one backup satellite in orbit at all times. This policy proved useful in December 2008 and again in September 2012, when the agency experienced problems with one of its operational satellites, but was able to move its backup satellite into place until the problems had been resolved.

However, NOAA is facing a period of up to 17 months when it will not have a backup satellite in orbit. Specifically, in April 2015, NOAA expects to retire one of its operational satellites (GOES-13) and to move its backup satellite (GOES-14) into operation. Thus, the agency will have only two operational satellites in orbit—and no backup satellite—until GOES-R is launched and completes an estimated 6-month post-launch test period. If GOES-R is launched in March 2016, the earliest it could be available for operational use would be September 2016. Figure 9 shows the potential gap in backup coverage, based on the launch and decommission dates of GOES satellites.

During the time in which no backup satellite would be available, there is a greater risk that NOAA would need to either rely on older satellites that are beyond their expected operational lives and may not be fully functional, request foreign satellite coverage, or to operate with only a single operational satellite. Agency officials stated that the risk of a gap may be reduced, because NOAA satellites have historically remained in operation longer than their expected life. While many satellites outlive their expected lifespans, the current GOES satellites are operating with reduced functionality, and one has experienced two major outages. Without a full complement of operational GOES satellites, the nation’s ability to maintain the continuity of data required for effective weather forecasting could be compromised. This, in turn, could put the public, property, and the economy at risk.

Any delay to the GOES-R launch date would extend the time without a backup to more than 17 months. As discussed earlier in this report, further delays to the committed launch date of the first GOES-R satellite are possible due to continued technical issues encountered during testing and integration.

GOES-R Satellite Contingency Plan Shows Improvement, but Continues to Lack Details in Key Areas

Government and industry best practices call for the development of contingency plans to maintain an organization’s essential functions—such as GOES satellite data—in the case of an adverse event. In September 2013, we reported on weaknesses in the contingency plans for NOAA’s satellites. At that time, we compared NOAA’s plans to 17 best practices associated with three main areas: identifying failure scenarios and impacts, developing contingency plans, and validating and implementing contingency plans. We reported that while NOAA identified failure scenarios, recovery priorities, and minimum levels of acceptable performance, the satellite contingency plan contained areas that fell short of best practices, such as working with the user community to account for potential reductions in capability under contingency operations. Furthermore, the agency did not identify alternative solutions or timelines for preventing a GOES-R launch delay.

In February 2014, NOAA released a new satellite contingency plan in response to these recommendations. This plan improved in comparison to many, but not all, of the best practices. Specifically, the plan improved in 6 areas and stayed the same in 4 areas. Table 10 compares our assessment of the current satellite contingency plan with our September 2013 analysis for all best practices that were not fully met.

Program officials stated that it is not feasible to include strategies to prevent delays in launch of the first GOES-R satellite in the contingency plan, because such strategies are not static. They explained that options for preventing a delay vary greatly over time based on issues that occur during the satellite’s development. They further stated that the program’s monthly presentations to NOAA management provide a summary of current threats to the launch date and strategies to mitigate those threats. For example, NOAA is considering whether or not to remove or delay selected ground system functions to post-launch, and the program provides monthly updates on this issue.

While actively managing the program to avoid a delay is critical, it is also important that NOAA management and the GOES-R program consider and document feasible alternatives for avoiding or limiting such a launch delay. This will allow stakeholders throughout NOAA to be aware of, respond to, and plan for the potential implementation of each alternative, not only the small number of alternatives the program is actively considering in any given month. Until NOAA addresses the remaining shortfalls in its GOES-R gap mitigation plan, the agency cannot be assured that it is exploring all alternatives or that are able to effectively prepare to receive GOES information in the event of a failure.

Conclusions

After spending 10 years and just over $5 billion, the GOES-R program is nearing the launch of its first satellite. However, it continues to face challenges in maintaining its schedule and controlling its costs. The program continues to experience delays in remaining major milestones, which could result in further delays to the launch date. Costs are increasing faster than expected for key program components and contractor data is often inconsistent from month to month. Until the agency ensures its contractor cost data are consistent, it will be more difficult for managers and program officials to make financial projections and assess reserve needs and usage.

As the GOES-R program progresses through its testing and integration phase, it is essential that it is able to appropriately handle defects that arise during testing. While NOAA and its contractors have implemented a defect management process which is successful in many areas, there are shortfalls in how the program defines defects, monitors trends, and reports on defects and defect metrics. In particular, NOAA has not established a standard set of metric information for all defects, including the dates on which defects were identified and resolved, or an indication of a defect’s severity. Also, NOAA did not clearly define the type of issue that constitutes a defect. Furthermore, multiple defects remain open on some program components, including several that have remained open for more than six months. Until the program addresses these shortfalls and reduces the number of open defects, it may not have a complete picture of remaining issues and faces an increased risk of further delays to the GOES-R launch date.

NOAA could experience a gap in satellite data coverage if GOES-R is delayed further and one of the two remaining operational satellites experiences a problem. NOAA has made improvements to its satellite contingency plan, but the plan still does not sufficiently address mitigation options for a launch delay, potential impacts, or minimum performance levels. Until such information is available, it will be difficult to integrate mitigation efforts, or to coordinate with users in the event of a failure.

Recommendations for Executive Action

To address risks in the GOES-R program development and to help ensure that the satellite is launched on time, we are making the following four recommendations to the Secretary of Commerce. Specifically, we recommend that the Secretary of Commerce direct the NOAA Administrator to: investigate and address inconsistencies totaling hundreds of thousands of dollars in monthly earned value data reporting for the GLM and ABI instruments; address shortfalls in defect management identified in this report, including the lack of clear guidance on defect definitions, what defect metrics should be collected and reported, and how to establish a defect’s priority or severity; and reduce the number of unresolved defects on the GOES ground system and spacecraft.

In addition, because NOAA has not fully implemented our prior recommendation to improve its satellite gap mitigation plan, we recommend that the Secretary of Commerce direct the NOAA Administrator to: add information to the GOES satellite contingency plan on steps planned or underway to mitigate potential launch delays, the potential impact of failure scenarios in the plan, and the minimum performance levels expected under such scenarios.

Agency Comments and Our Evaluation

We sought comments on a draft of our report from the Department of Commerce and NASA. We received written comments from the Deputy Secretary of Commerce transmitting NOAA’s comments. NOAA concurred with all four of our recommendations and identified steps that it plans to take to implement them. It also provided technical comments, which we have incorporated into our report, as appropriate. NOAA’s comments are reprinted in appendix III.

On November 14, 2014, an audit liaison for NASA provided an e-mail stating that the agency would provide any input it might have to NOAA for inclusion in NOAA’s comments.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to interested congressional committees, the Secretary of Commerce, the Administrator of NASA, the Director of the Office of Management and Budget, and other interested parties. The report also will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions on the matters discussed in this report, please contact me at (202) 512-9286 or at pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) assess progress on the GOES-R program with respect to planned schedule, cost, and functionality; (2) assess efforts to identify and address issues discovered during integration and testing; and (3) evaluate the likelihood of a gap in satellite coverage and analyze the adequacy of contingency actions in place to prevent or mitigate such a gap.

To assess NOAA’s progress on the GOES-R program with respect to planned schedule, cost, and functionality, and to identify risks that could lead to further schedule delays, we analyzed data from monthly program management meetings. We evaluated progress made in completing key program components and major program reviews, and compared planned and actual completion dates for key program milestones over the last two to three years to determine the degree to which these dates have changed. To ensure that the program’s schedule data were consistent and reliable, we compared milestone data over several months and contacted agency officials to corroborate events that occurred over the course of our engagement. We analyzed earned value management (EVM) data to compare levels of cost variance over time for key program components, and calculated earned value management metrics using program cost performance reports. To ensure that the program’s cost data were reliable, we compared formulas from the GAO Cost Guide to the program’s EVM approach. We also compared EVM data across a series of monthly program cost performance reports. In doing so, we found inconsistencies in the monthly EVM data for selected components, and reported on those inconsistencies in this report. However, the data were sufficient for our purpose of assessing overruns because these inconsistencies were small in comparison to cost variances. We assessed recently enacted and potential changes in functionality for key program components. We also interviewed program officials regarding changes in schedule milestones, cost performance and reserve funding, and functionality.

To assess efforts to identify and address issues discovered during integration and testing, we identified best practices in defect management from leading industry and government organizations. We compared NOAA policy documents and defect management artifacts by the GOES program, its contractors, and an independent NOAA mission assurance group to the best practices. We selected three components, which are critical to the program’s mission requirements, and evaluated recent test results and defects. Specifically, we identified two recent tests for each component and analyzed artifacts associated with defects identified during those tests. We compared the defects against each best practice to determine if NOAA and its contractors fully implemented, partially implemented, or did not implement the best practices. The agency and contractor had to meet all aspects of the best practice to achieve a fully implemented score, some aspects to achieve a partially implemented score, and no aspects to achieve a not implemented score. To identify the program’s recent performance in closing and managing defects, we analyzed basic defect trend information—such as number of defects opened and closed, and defect severity—for each of several program components. We developed charts to demonstrate defect trends. To ensure the data were reliable, we compared defect data from individual defect reports to agency trend charts, and sought corroboration from agency and contractor officials. We also interviewed agency and contractor officials to discuss their defect management processes and practices, and to confirm information we found while analyzing individual defect reports and trend charts.

To evaluate the likelihood of a gap in satellite coverage, we reviewed monthly program management presentations and other review board documentation. To analyze the adequacy of contingency actions in place to prevent or mitigate such a gap, we compared NOAA’s latest satellite contingency plan to best practice criteria from industry and government. We focused specifically on the ten areas we identified as weak in our prior report. For each of the ten areas, we rated NOAA’s contingency plan as having partially or fully implemented the best practice criteria. In addition, we interviewed agency officials regarding the likely amount of a gap in on-orbit backup coverage of GOES satellites, the potential for a gap in operational coverage, and efforts to improve the GOES contingency plan.

We conducted this performance audit from January 2014 to December 2014, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Defect Trends for Key GOES-R Components

As noted earlier, the GOES-R program does not obtain or maintain defect trend data for the program as a whole; however, data on individual components and portions of components show that a large number of defects remain open, including several high-priority defects. The following sections provide more details on recent data trends for the GOES-R ground system and spacecraft as well as the ABI and GLM instruments. It is important to note that the sections do not provide consistent information among components because the program does not require consistent metrics and the contractors document different data.

Ground System Has Many Unresolved Defects

As of the end of September 2014, the ground system had 500 open defects, an increase from 342 open defects at the end of September 2013. Also, as of the end of September 2014, 96 defects that were identified prior to January 2014 remained open, and 36 high-priority defects (those rated as critical or moderate) remained open.

Program and contractor officials provided rationale and insight into the mitigating circumstances surrounding these defects. Program officials stated that the number of open defects during the integration and test phase is as expected for a ground system of a magnitude similar to GOES. Contractor officials reported that none of the 96 longstanding defects were in either of the two highest severity categories. They also noted that most of the open high-priority defects had been opened during testing events conducted over the previous 3 months, which is a reasonable length of time to close a defect.

Furthermore, according to contractor officials, many of the longstanding open defects were in categories such as documentation-related defects which are considered less severe. Officials also stated that defects in these categories are often kept open for specific reasons, such as to gain cost efficiencies or to wait for an already-planned test event rather than creating a new test event. However, as of September 2013, 193 of the 342 defects were not related to documentation and this number rose to 416 of the 500 open defects in September 2014. Figure 10 shows the increase in open defects in the period from September 2013 to August 2014 for the GOES-R ground system.

As allowed by NOAA’s guidance, contractors on the spacecraft track defects differently than the ground system contractors do. Thus, it is not possible to compare opened and closed defects over time as depicted above for the ground system. Instead, spacecraft contractors track hardware and software defects independently. They also track when new defects are identified and how long they stay open.

For spacecraft hardware, the defect data show that the average time it took to resolve defects increased over time. Specifically, the average age for hardware defects increased from 99 days in May 2013 to a high of 167 days in April 2014, at which point 58 percent of all open defects had been open for at least 90 days. Program officials stated that they began continuous 24-hour a day, 7-day a week testing in October 2014, which means that they should be able to make progress in closing the remaining defects.

While the backlog of hardware-related defects on the spacecraft remains high, NOAA has been effective in recent months in greatly reducing the number of software-related defects on the spacecraft. The total number of open software-related defects increased and remained high through February 2014, but then declined significantly in March and April 2014.

Defect Trends Show Decline in Defects for GLM Instrument

Defect data for the GLM instrument showed a decline in the total number of defects for both hardware and software components. While there were 117 unresolved hardware defects in May 2014, only 13 hardware defects remained unresolved as of September 2014. Trend data also showed a decline in the number of unresolved software defects. Specifically, the total number of software-related defects remaining open never increased above 7 over the period from March 2013 to January 2014. For half the year, there was no more than one open defect.

ABI Hardware and Software Defects Were Closed in Recent Months

Metrics for the ABI instrument show that all hardware defects and all but one software defect have been closed, likely because the first unit of the instrument was completed. The number of hardware defects occurring each month declined to near zero during the period February 2013 to April 2014. Less than ten ABI software defects were open at any point from September 2012 onward, and only 11 defects were newly opened during that time. Figure 11 shows the number of opened and closed ABI hardware defects by month, and figure 12 shows opened and closed defects for ABI software.

Appendix III: Comments from the Department of Commerce

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, individuals making contributions to this report included Colleen Phillips (assistant director), Alexander Anderegg, Christopher Businsky, Shaun Byrnes, James MacAulay, and Karl Seifert.