Geostationary Operational Environmental Satellites: Further Actions Needed to Effectively Manage Risks
Released: 2007-10-23T00:00:00

Letter

Background

Since the 1960s, geostationary and polar-orbiting environmental satellites have been used by the United States to provide meteorological data for weather observation, research, and forecasting. NOAA’s National Environmental Satellite Data and Information Service (NESDIS) is responsible for managing the civilian geostationary and polar-orbiting satellite systems as two separate programs, called GOES and the Polar Operational Environmental Satellites, respectively.

Unlike polar-orbiting satellites, which constantly circle the earth in a relatively low polar orbit, geostationary satellites can maintain a constant view of the earth from a high orbit of about 22,300 miles in space. NOAA operates GOES as a two-satellite system that is primarily focused on the United States. These satellites are uniquely positioned to provide timely environmental data to meteorologists and their audiences on the earth’s atmosphere, its surface, cloud cover, and the space environment. They also observe the development of hazardous weather, such as hurricanes and severe thunderstorms, and track their movement and intensity to reduce or avoid major losses of property and life. Furthermore, the satellites’ ability to provide broad, continuously updated coverage of atmospheric conditions over land and oceans is important to NOAA’s weather forecasting operations.

To provide continuous satellite coverage, NOAA acquires several geostationary satellites at a time as part of a series and launches new satellites every few years (see table 1).

Three satellites—GOES-11, GOES-12, and GOES-13—are currently in orbit. Both GOES-11 and GOES-12 are operational satellites, while GOES-13 is in an on-orbit storage mode. It is a backup for the other two satellites should they experience any degradation in service. The others in the series, GOES-O and GOES-P, are planned for launch over the next few years. NOAA is also planning the next generation of satellites, known as the GOES-R series, which are planned for launch beginning in 2014.

GOES-R Program—An Overview

NOAA plans for the GOES-R program to improve on the technology of prior series, in terms of both system and instrument improvements, to fulfill more demanding user requirements and to provide more rapid information updates. Table 2 highlights key system-related improvements GOES-R is expected to make to the geostationary satellite program.

In addition to the system improvements, the instruments on the GOES-R series are expected to significantly increase the clarity and precision of the observed environmental data. NOAA originally planned to acquire six different types of instruments. Furthermore, two of these instruments— the Advanced Baseline Imager and the Hyperspectral Environmental Suite—were considered to be the most critical because they would provide data for key weather products. Table 3 summarizes the originally planned instruments and their expected capabilities.

More recently, however, NOAA reduced the scope of the GOES-R program because of expectations of higher costs. In May 2006, the program office projected that total costs, which were originally estimated to be $6.2 billion, could reach $11.4 billion. We reported that this led NOAA to reduce the scope and technical complexity of the baseline program. Specifically, in September 2006, NOAA reduced the minimum number of satellites from four to two, cancelled plans for developing the Hyperspectral Environmental Suite, and estimated the revised program would cost $7 billion. Table 4 provides a summary of the timeline and scope of these key changes.

GOES-R Program Office Structure

NOAA is solely responsible for GOES-R program funding and overall mission success. However, since it relies on NASA’s acquisition experience and technical expertise to help ensure the success of its programs, NOAA implemented an integrated program management structure with NASA for the GOES-R program. Within the program office, there are two project offices that manage key components of the GOES-R system. These are called the flight and operations project offices. The flight project office oversees the spacecraft, instruments, and launch services. The operations project office oversees the ground elements and on-orbit operations of the satellites. The project manager for the flight project office and the deputy project manager for operations project office are designated to be filled with NASA personnel. Additionally, NOAA has located the program office at NASA’s Goddard Space Flight Center.

Planned GOES-R Acquisition Strategy

NOAA’s acquisition strategy was to award contracts for the preliminary design of the GOES-R system to several vendors who would subsequently compete for the contract to be the single prime contractor responsible for overall system development and production. As such, in October 2005, NOAA awarded contracts for the preliminary design of the overall GOES-R system to three vendors.

In addition, to reduce the risks associated with developing technically advanced instruments, NASA awarded contracts for the preliminary designs for five of the originally planned instruments. NASA expected to subsequently award development contracts for these instruments and to eventually turn them over to the prime contractor responsible for the overall GOES-R program.

GOES-R Preliminary Design Studies Are Completed, but Key Program Changes Have Been Made and Cost and Schedule Estimates Are Likely to Grow

NOAA has completed preliminary design studies of its GOES-R procurement. In addition, the agency recently decided to separate the space and ground elements of the program into two separate contracts to be managed by NASA and NOAA, respectively. However, this change has delayed a key decision to proceed with the acquisition, which was planned for September 2007. Further, independent estimates are higher than the program’s current $7 billion cost estimate and convey a low level of confidence in the program’s schedule for launching the first satellite by 2014. As NOAA works to reconcile the independent estimate with its own program office estimate, costs are likely to grow and schedules are likely to be delayed.

Progress Has Been Made on GOES-R Procurement Activities

NOAA and NASA have made progress on GOES-R. The program office has completed preliminary design studies of the overall GOES-R system and has initiated development work on most of the planned instruments. Specifically, the NOAA-issued contracts for the preliminary design of the overall GOES-R system to three vendors have ended, and the designs have been completed.

In addition, after completing preliminary designs on five of the originally planned instruments, NASA awarded development contracts for three of them. Further, the most critical of these instruments—the Advanced Baseline Imager—has completed a major development milestone. In February 2007, it passed a critical design review gate and NASA approved the contractor to begin production of a prototype model.

NOAA Revised its Acquisition Strategy

NOAA recently made a number of key changes in how it plans to acquire the GOES-R system. Originally, NOAA planned to award and manage a single prime contract for the acquisition and operation of the integrated system. However, an independent review team assessed the program and found that this approach was risky. It recommended that NOAA split the acquisition effort into two separate contracts for the space and ground segments and have NASA manage the space segment. The independent review team concluded that there was less risk in continuing with this approach than there would be if NOAA took on a new and expanded role.

In March 2007, Commerce approved NOAA’s decision to implement these recommendations. The agency revised its acquisition strategy to include two separate contracts—the space segment and the ground segment. The two contracts are expected to be awarded in May 2008 and August 2008, respectively. The space segment is to be managed by a NASA-led flight project office. As such, NASA is to be responsible for awarding and managing the space segment contract, delivering the flight-ready instruments to the space segment contractor for integration onto the satellites, and overseeing the systems engineering and integration. NOAA is to be responsible for the ground segment contract, which is to be managed by the NOAA-led operations project office.

The revised acquisition strategy has delayed NOAA’s plans to complete a key decision milestone on whether to proceed with GOES-R development and production in September 2007. Once this decision is made, the final requests for proposals on the system segments are to be released. The agency could not provide a timeframe for when this key decision milestone would take place.

GOES-R Cost Estimates Are Likely to Grow and Schedule Estimates Are Likely to Slip

NOAA’s current estimate that the life cycle cost of the GOES-R program would be $7 billion is likely to grow, and its estimate that the first satellite would be launched in December 2014 is likely to slip. Consistent with best practices in cost estimating, in May 2007, NOAA had two different cost estimates completed for the current GOES-R program—one by its program office and one by an independent cost estimating firm. The program office estimated with 80 percent confidence that the program would cost $6.9 billion. The independent estimating firm estimated with 80 percent confidence that the program would cost $9.3 billion.

A comparison of the two cost models shows that the independent estimator has about a 20 percent level of confidence that the program can be completed for $6.9 billion. Further, the independent estimator concluded that the program office estimate significantly understated the risk of cost overruns. Other major differences between the two estimates are contained in government costs and in the space and ground segments. In commenting on a draft of the accompanying report, NOAA officials noted that one of the differences between the estimates is the inflation rate. The independent estimator assumed a higher inflation rate than the rate that NOAA and NASA typically use. NOAA officials noted that if the independent estimate was adjusted to NOAA’s inflation rate, the program’s cost estimate—with 80 percent confidence—would be $8.7 billion. However, we believe that the value of an independent estimate is that it does not necessarily use the same assumptions as the program office. By offering alternative assumptions, the independent estimate provides valuable information for government officials to consider when revising program cost estimates.

Program officials are reconciling the two different cost estimates and plan to establish a new program cost estimate to be released in conjunction with the President’s fiscal year 2009 budget in February 2008. Program officials were unable to provide us information on the reconciled estimate until it is released. Nonetheless, the revised cost estimate will likely be $1 billion more than the current $7 billion.

Regarding schedule, NOAA’s current plan to launch the first GOES-R series satellite in December 2014 could be delayed. This schedule was driven by a requirement that the satellites be available to back up the last remaining GOES satellites (GOES-O and GOES-P) should anything go wrong during the planned launches of these satellites (see table 5). However, as part of its cost estimate, the independent estimator performed a schedule risk analysis. The independent estimator determined that there was less than a 50 percent chance that the first satellite would be ready for launch by December 2014 and that a later date would be more realistic. The estimator determined that it had 50 percent confidence that the first satellite would launch by October 2015 and 80 percent confidence that the satellite would launch by March 2017. A delay of this magnitude could affect the continuity of GOES data should the agency experience problems with the predecessor satellites.

NOAA is Taking Steps to Address Key Risks, but More Remains to Be Done

To address cost, schedule, and technical risks, the GOES-R program established a risk management program and has taken steps to identify and mitigate selected risks. However, more remains to be done to fully address a comprehensive set of risks. Specifically, the program has multiple risk watchlists and they are not always consistent. Further, key risks are missing from the risks lists, including risks associated with unfilled executive positions, limitations in NOAA’s insight into NASA’s deliverables, and insufficient funding for unexpected costs (called management reserve) on a critical sensor. As a result, the GOES-R program is at increased risk that problems will not be identified or mitigated in a timely manner and that they could lead to program cost overruns and schedule delays.

GOES-R Has a Risk Management Program and Is Taking Measures to Address Selected Risks

The GOES-R program office established a risk management program and is tracking and mitigating selected risks. Risk management is a leading management practice that is widely recognized as a key component of a sound system development approach. An effective risk management approach typically includes identifying, prioritizing, and mitigating risks, and escalating key risks to the attention of senior management.

In accordance with leading management practices, the GOES-R program identifies risks, assigns a severity rating to risks, tracks these risks in a database, plans response strategies for each risk in the database, and reviews and evaluates these risks during monthly program risk management board meetings. Programwide and project-specific risks are managed by different offices. The program office identifies and tracks programwide risks—those that affect the overall GOES-R program. NASA’s flight project office and NOAA’s operations project office manage risks affecting their respective aspects of the program. Further, the program office briefs senior executives on top program and project risks on a monthly basis.

GOES-R Program Office Identified and Is Working to Mitigate Programwide Risks

As of July 2007, the program office identified three program risks affecting the overall GOES-R program. These risks include the development of the integrated master schedule, the ability to secure authorization to use a key frequency band to meet the space-to-ground communication data link requirements for the GOES-R system, and the final approval of the GOES-R mission requirements from the NOAA Deputy Undersecretary.

NOAA is working to mitigate and close program risks that it is tracking. For example, the program office recently closed the risk associated with GOES-R requirements because it had sufficiently defined and obtained approval of these requirements. As another example, the program office considers the lack of an integrated master schedule to be its highest priority risk. Program officials reported that completion of the integrated master schedule is driven by the completion of the intermediate schedules for the ground segment and the space-to-ground interdependencies. Key program staff members, including a resident scheduler, meet on a weekly basis to resolve outstanding design issues and hone these schedules. Program officials reported that the intermediate schedules are near completion and that they plan to have the integrated master schedule completed in Fall 2007. They expect to remove this issue from the risk watchlist at that time.

NASA Identified Flight Segment Risks and Is Working to Mitigate Them

As of July 2007, the NASA flight project office identified four risks affecting instrument development, all of which are classified as medium risk. The top three risks pertain to the advanced imaging instrument, ABI—including issues on timely and quality subcontractor delivery of a critical part, stray light negatively impacting the performance of the optical system, and meeting specified performance requirements on image navigation and registration. The fourth priority risk pertains to the improvement of subcontractor quality assurance on a key sensor for the Space Environmental In-Situ Suite.

NASA is working to mitigate the flight segment risks that it is tracking. For example, the ABI contractor, among other things, plans to complete a key simulation review before the end of the year (called the structural thermal optical performance analysis) to evaluate whether the instrument can meet its expected performance parameters for image navigation and registration. NASA also recently conducted a vendor facility assessment of the Space Environmental In-Situ Suite subcontractor to determine whether adequate quality assurance improvements had been made to be compliant with contract requirements. These actions are expected to help mitigate the risk.

NOAA Identified Risks in its Operations Segment and Is Working to Mitigate Them

As of July 2007, the NOAA operations project office identified five risks impacting the management and development of the ground system and operations, including one that is identified as a medium risk. These risks include, among other things, inadequate definition of flight and operations project interdependencies, algorithm development responsibilities, and the adequate definition of coordination requirements between the space and ground segments to ensure that the two requests for proposals are consistent.

NOAA is working to mitigate the ground system and operations risks that it is tracking. For example, for the highest priority risk regarding schedule interdependencies, key staff from both the flight and operations projects meet weekly in order to identify and synchronize project schedules. The project office expects to close this risk in Fall 2007.

Multiple Watchlists Are Not Consistent, Making it Difficult to Prioritize and Manage Risks

While GOES-R has implemented a risk management process, its multiple risk watchlists are not consistent in areas where there are interdependencies between the lists, which makes it difficult to effectively prioritize and manage risks at the appropriate organizational levels. Sound risk management practices call for having a consistent prioritization approach and for significant problems to be elevated from the component level to the program level. This is because an issue affecting a critical component could have severe programmatic implications and should be identified, tracked, and overseen at the program level. In addition, program executives should be briefed regularly on the status of key risks.

However, on the GOES-R program, the risks identified on the multiple risk lists are inconsistent in areas where there are interdependencies between the lists. These interdependencies include situations where a risk is raised by one project office and affects the other project office, but is not identified by the other project office or elevated to the program level risk list. They also include situations where a risk identified by a project office has programwide implications, but is not elevated to the program level risk list. For example, the operations project office identified schedule interdependencies between the flight and operations project offices as a medium criticality risk, but neither the flight project office nor the program identified this risk even though it is relevant to both. As another example, the operations project office identified the ground procurement schedule as a major issue in its briefing to senior management, but this risk was not identified on its own or on the programwide risk lists.

In addition, while the three offices brief senior management about their key risks on a monthly basis, selected risks may not be accurately depicted in these briefings because of the inconsistencies among the risk watchlists. For example, both the flight and operations project offices identified technical development issues as minor to moderate risk areas, but the program office did not identify this item as a risk and, when it briefed senior management, it noted that technical development was in good shape. Figure 1 depicts examples of inconsistencies among risk lists and briefings to senior management.

The lack of consistency in managing risks in areas where there are interdependencies makes it difficult to ensure that all identified risks are appropriately prioritized and managed. This situation hampers the program office’s ability to identify and mitigate risks early on and to anticipate and manage the impact of risks on other areas of the program.

Important GOES-R Management Risks Are Missing from the Program Watchlist

To be effective, a risk management program should have a comprehensive list of risks. However, several key risks that impact the GOES-R procurement and merit agency attention are not identified in the program’s risk lists. These risks include (1) key leadership positions that need to be filled, (2) NOAA’s limited insight into NASA’s deliverables, and (3) insufficient management reserves (held by the program and a key instrument contractor). At the conclusion of our review for the accompanying report, program officials stated that they are aware of these issues and are working to monitor them or address them, as warranted. Nevertheless, until these and other programwide risks are identified and addressed as part of a comprehensive risk management program, there is increased likelihood that issues will be overlooked that could affect the acquisition of the GOES-R system.

Key GOES-R Leadership Positions Need to be Filled

The two senior GOES-R program positions—the system program director and deputy system program director—are currently filled by NASA and NOAA personnel in an acting capacity until they can be permanently filled by NOAA. In addition, the acting system program director is not able to work full time in this role because she is also on a special assignment as the NESDIS Deputy Assistant Administrator for Systems. NOAA reported that it plans to fill the deputy system program director role in the near future, but noted that it could take more than 6 months to fill the system program director role. Given the approach of the development phase of the GOES-R acquisition and the competing priorities of the acting system program director, it is especially important that these key leadership positions be filled quickly. At the conclusion of our review, agency officials stated that they are aware of this issue and are working to fill the positions, but they did not believe the issue warranted inclusion on the program level risk watch list. However, without the senior level attention inherent in a sound risk management program, it is not clear that NOAA is sufficiently focused on the importance of establishing knowledgeable and committed program executives, or in moving quickly to fill these critical positions.

NOAA’s Insight into NASA’s Program Elements Is Limited

NOAA’s March 2007 decision to adopt an acquisition management approach similar to prior GOES procurements could make the agency vulnerable to repeating some of the problems experienced in the past. In particular, our work on the GOES I-M series found that NOAA did not have the ability to make quick decisions on problems because portions of the procurement were managed by NASA. In fact, NOAA officials originally intended to depart from this approach as a lesson they learned from the GOES I-M acquisition, because it limited the agency’s insight and management involvement in the procurement of major elements of the system.

The established NOAA/NASA interagency agreements require NASA to submit monthly contractor cost performance reports to NOAA and to alert NOAA should cost and schedule performance drop below certain thresholds. NASA is currently submitting the required reports and has alerted NOAA on major cost and schedule changes. However, these interagency agreements do not contain provisions that enable NOAA to ensure that the data and reports are reliable and that they accurately depict contractor performance. To do so would entail NOAA having the ability and means to question and validate data, such as by having direct access to the contractor.

NASA and NOAA officials reported that the two agencies are working together with an unparalleled level of transparency and noted that NOAA program staff have access to contractor data and can bring any questions with the data to the relevant NASA staff. However, they acknowledged that this process is not documented and were not able to demonstrate that NOAA staff had questioned contract data and that NASA had facilitated obtaining answers to the questions. By not identifying and mitigating this risk on its program risk list, NOAA increases the likelihood that the GOES- R program will repeat the management and contractor shortfalls that plagued past GOES procurements.

Recent Changes on a Key Instrument Have Reduced Program Management Reserve Funds and Limited Contractor Reserve Funds Leave GOES-R Vulnerable to Future Cost Increases

A recent modification to the critical ABI instrument contract increased its cost, thereby reducing the amount of management reserve funds held by the program office for unexpected expenses. In September 2006, we reported that ABI was experiencing technical challenges, that were resulting in cost and schedule overruns. Since then, the contractor continued missing cost and schedule targets—a trend that continued until February 2007. At that time, NASA modified the contract to implement a revised baseline cost and schedule. The added cost of this modification was funded using management reserve funds held by the GOES-R program office. As a result, the amount of reserve held by the program office dropped below 25 percent—a level that NOAA reported it intended to establish as a lesson learned from other satellite acquisitions. As of July 2007, the program’s reserve level was at about 15 percent. Program officials stated that their revised goal is to maintain between 10 and 15 percent in reserve at the program level. While maintaining a 10 to 15 percent management reserve is on par with other major satellite acquisitions, the depletion of management reserves this early in the GOES- R acquisition raises concerns that there will be insufficient reserves during the challenging development, integration, and testing phases to come.

In addition, the contractor for the ABI instrument has a very low level of reserve funding for unexpected costs, which means that any unexpected problems will likely lead to cost growth on the overall GOES-R program. As of May 2007, the contractor was holding less than 1 percent of funding in reserve to cover unexpected costs associated with the 40 percent of work left to be completed. As such, there is a risk that the new baseline could fail due to inadequate reserves to finish the program. This would likely have a diminishing effect on the reserve held by the GOES-R flight project and the program office to cover the costs of a second revised baseline plan. Our prior work on system acquisitions has shown inadequate reserves to be an indicator of poor management performance that could lead to cost overruns. Considering that GOES-R has not yet entered the development and production phases, it will be critical for NOAA’s senior executive management to aggressively manage this risk. By not identifying, mitigating, and tracking this risk in a programwide risk list, the GOES-R program runs an increased risk that unanticipated issues on the ABI instrument will lead to programwide cost overruns and schedule delays.

Implementation of GAO Recommendations Should Improve NOAA’s Ability to Effectively Manage the GOES-R Procurement

To improve NOAA’s ability to effectively manage the procurement of the GOES-R system, we recommended in our accompanying report that the Secretary of Commerce direct the Undersecretary of Commerce for Oceans and Atmosphere to take the following two actions: Ensure that the GOES-R program office manages, mitigates, and reports on risks using a program-level risk list that is reconciled with and includes risks from its flight and operations project offices that could impact the overall program.

Include the following risks on the programwide risk list, develop plans to mitigate them, and report to senior executives on progress in mitigating them: unfilled or temporary GOES-R program leadership positions, insufficient program insight on NASA contract performance, and insufficient management reserve on the critical Advanced Baseline Imager instrument and at the GOES-R program level.

In written comments, Commerce agreed with our recommendations to use a program level risk list and to add selected risks to its list. The department reported that NOAA has established a consolidated programwide risk list that is to be used to evaluate risks during monthly internal and external reviews. Further, NOAA acknowledges the risks associated with having unfilled leadership positions and insufficient management reserves and is working to mitigate these risks. However, the department disagreed with our recommendation to manage and mitigate the risk that NOAA has insufficient insight into NASA’s contracts. The department cited an unparalleled level of transparency between the two agencies and listed multiple regular meetings that the two agencies hold to ensure close coordination. While an improved working relationship between the two agencies is critical, NOAA has not provided any evidence that it has been able to effectively question and validate data on NASA’s contractor performance. Given the past problems that NOAA has experienced in obtaining insight into NASA’s contracts and the importance of this interagency relationship to the success of the GOES-R program, we believe that this issue should be managed and monitored as a risk.

NOAA also requested that we acknowledge its effort to reconcile its program estimate with the independent estimate and reflect a 20 percent possibility that the program could cost $1 billion more than the current estimate of $7 billion, rather than $2 billion more. We acknowledge this in our report; however, the reconciliation effort is not complete and NOAA did not provide us with a reconciled estimate.

In summary, although NOAA has made progress in the GOES-R procurement, changes in the GOES-R acquisition strategy could lead to cost overruns and schedule delays if not managed effectively. Over the last year, NOAA has completed preliminary design studies of its GOES-R system and decided to separate the space and ground elements of the program into two contracts and have NASA oversee the system integration effort. Current program plans call for a two-satellite program—estimated to cost about $7 billion—with launch of the first satellite in December 2014. However, independent studies show that the program’s cost could increase by about $2 billion and that the first launch could be delayed by at least 2 years.

NOAA has taken steps to identify and address key risks but more could be done to effectively manage risks from a programwide perspective. In particular, the program has multiple risk watchlists that are not consistent in areas where there are interdependencies and key risks have not been elevated for programwide attention. Also, several risks that warrant NOAA’s attention have not been placed on any watchlist. Specifically, the top two leadership positions are only temporarily filled; NOAA does not have the ability and means to obtain insight into NASA contracts in order to validate contractor performance data; and insufficient management reserves to handle unexpected problems on a critical instrument and at the program level are likely to affect overall program costs when any unexpected problems arise. Until NOAA manages and addresses a comprehensive set of program risks, the agency’s ability to effectively manage the GOES-R acquisition will be significantly weakened and could lead to substantial program overruns and delays.

Mr. Chairman, this concludes my statement. I would be happy to answer any questions that you or members of the subcommittee may have at this time.

If you have any questions on matters discussed in this testimony, please contact me at (202) 512-9286 or by e-mail at pownerd@gao.gov. Other key contributors to this testimony include Carol Cha, Neil Doherty, Nancy Glover, Colleen Phillips (Assistant Director), and Teresa Smith.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.