Geostationary Weather Satellites: Design Progress Made, but Schedule Uncertainty Needs to be Addressed
Released: 2012-06-27T00:00:00

Letter

Background

Since the 1960s, geostationary satellites have been used by the United States to provide meteorological data for weather observation, research, and forecasting. NOAA’s National Environmental Satellite, Data, and Information Service is responsible for managing the civilian operational geostationary satellite system, called GOES. Geostationary satellites can maintain a constant view of the earth from a high orbit of about 22,300 miles in space.

NOAA operates GOES as a two-satellite system that is primarily focused on the United States (see fig. 1). These satellites provide timely environmental data about the earth’s atmosphere, surface, cloud cover, and the space environment to meteorologists and their audiences. They also observe the development of hazardous weather, such as hurricanes and severe thunderstorms, and track their movement and intensity to reduce or avoid major losses of property and life. The ability of the satellites to provide broad, continuously updated coverage of atmospheric conditions over land and oceans is important to NOAA’s weather forecasting operations.

To provide continuous satellite coverage, NOAA acquires several satellites at a time as part of a series and launches new satellites every few years (see table 1).satellites and one backup satellite in orbit at all times.

Progress Made in Completing Early Design, but Milestones Were Completed Late and Development Costs Have Increased

While the GOES-R program has made progress in completing its early design and is nearing the end of the design phase for its flight and ground system components, it completed key design milestones later than planned. Recent technical problems with the instruments and spacecraft, as well as a significant modification to the ground project’s development plan, have delayed the completion of key reviews and led to increased complexity for GOES-R’s development. Moreover, several instrument, spacecraft, and ground system problems identified during design, have not yet been resolved. In addition to the delays, the technical and programmatic challenges experienced by GOES-R’s flight and ground projects have led to increased costs for its development contracts. Despite these problems, program officials report that planned launch dates and cost estimates for the first two satellites have not changed, and that approximately $1.2 billion is currently in reserve to manage future delays and cost growth. However, the program has used approximately 30 percent of its reserves over the last 3 years and significant portions of development remain for major components—including the spacecraft and Core Ground System. In addition, the program did not change its reserves when it restored two satellites adding approximately $3.2 billion to the program’s baseline. As a result, the program will be challenged in completing its remaining development, particularly the final design and testing of the spacecraft and ground system, within its cost and schedule targets.

GOES-R Has Made Progress on Its Design, but Initial Design Milestones Completed Later than Planned

Two key types of development milestones, which are identified in the GOES-R December 2007 management control plan, are the preliminary design review (PDR) and a more detailed critical design review (CDR). The PDR is an initial design milestone that assesses the readiness of the program to proceed with detailed design activities and the CDR is intended to demonstrate that the design is complete and appropriate to support proceeding to full-scale software development, as well as fabrication, assembly, integration, and testing. The program planned to complete the flight project’s PDR in April 2010 and CDR in April 2011. It planned to complete the ground project’s PDR in July 2010 and CDR in July 2011. These project-level reviews are to be completed before the program’s comprehensive PDR and CDR can be completed.

The program has demonstrated progress toward completing its design. Specifically, the program and its subsequent projects have successfully completed their PDRs, demonstrating that they are ready to proceed with detailed design activities. The program and its projects are also currently progressing towards the final design for the entire GOES-R system, which is expected to be completed at the program’s CDR planned for August 2012.

However, the program’s PDR milestones were completed later than their 2007 planned dates. Consequently, its CDR milestones were similarly delayed. Table 4 highlights delays in key program milestones as of April 2012.

Going forward, assembly and testing of all flight instruments for the first satellite in the series is to be completed by August 2013. Both the flight and ground projects’ development components are expected to be complete by September 2015. Despite recent delays in program milestones, NOAA still expects to meet an October 2015 launch date for the first satellite in the series by utilizing planned schedule reserves.

The Flight Project Office has made progress by completing the critical design reviews for each of its five main instruments, which is significant because the instrument designs will be applied to each satellite in the GOES-R series. The Geostationary Lightning Mapper was the most recent instrument to complete its critical design review, which occurred in August 2011—16 months after its planned completion date. Instrument fabrication, assembly, and testing activities are under way. The final instrument scheduled for completion—the Geostationary Lightning Mapper—is expected to be delivered for integration with the spacecraft by August 2013.

Although instrument design milestones are complete and assembly and testing activities are under way, each of the instruments and the spacecraft has recently encountered technical challenges. The Flight Project Office has taken steps to resolve several technical problems through additional engineering support and redesign efforts. However, there are still important technical challenges to be addressed, including signal blurring problems for several of the Advanced Baseline Imager’s infrared channels and Geostationary Lightning Mapper emissions that are exceeding specifications. The project office is monitoring these problems and has plans in place to address them. The current development efforts and recent challenges for components of the flight project are described in table 5.

The Ground Project Office has made progress in completing the ground system’s preliminary design. However, in doing so it experienced problems in defining ground system software requirements and identified problems with its dependencies on flight project schedules. Specifically, the project office discovered in early 2011 that software design requirements had not progressed enough to conduct the ground system’s preliminary design review. In addition, the ground system’s development schedule included software deliveries from flight project instruments that were not properly integrated—they had not yet been defined or could not be met. To address these problems, the Ground Project Office made significant revisions to the Core Ground System’s baseline development plan and schedule.

In order to avoid potential slippages to GOES-R’s launch date, project officials decided to switch from plans to deliver software capabilities at major software releases to an approach where software capabilities could be delivered incrementally (prior to major releases) as the project received data inputs from the flight project. According to NOAA and program officials, the revised development approach is to provide flexibility in the ground system’s development schedule and reduce risk associated with the original waterfall schedule. However, the revised plan is expected to cost $85 million more than the original plan through the Core Ground System’s completion. This cost includes increased contractor and government staff, new oversight tools, and more verification and testing activities associated with an increased number of software deliveries. Program officials acknowledged that the revised plan intends to expend additional resources to reduce schedule risk and potential impacts to GOES-R’s launch date, but that it also introduces new cost and schedule risks associated with incremental development, such as more software development and verification activities that require additional government oversight and continuous monitoring.

In addition, the program has cancelled previously exercised options to the ground project that were once considered part of its original baseline. In early 2011, the program determined that it could no longer fund Core Ground System contract options—which were estimated to cost approximately $50 million. Program officials stated that they cancelled the contract options due to approaching development commitments, including revisions to the ground system’s revised plan and schedule, and funding reductions from fiscal year 2011. According to program officials, the work to be performed under the cancelled contract options could be addressed by NOAA after GOES-R satellites are launched; however, there are currently no plans in place to do so.

Table 6 describes the development efforts and recent challenges for ground project components.

The GOES-R program’s estimated costs include, among other things, actual and estimated contract costs associated with design, development, integration, and testing activities for the instruments, spacecraft, and ground system as well as procurement of the satellites’ launch vehicles. The estimated costs also include government costs such as NOAA and NASA program management support and contingency reserves to be applied towards critical risks and issues as they arise. As of January 2010, the program reported estimates of $3.3 billion for the flight project (including reserves of $598 million), $1.7 billion for the ground project (including reserves of $431 million), $2.0 billion for other program costs (including reserves of $617 million), and $748 million for operations and maintenance.

Although NOAA has not changed its program cost estimates for the development of GOES-R and GOES-S, contract costs for the instruments, spacecraft, and ground system are rising. Specifically, contractor estimated costs for flight and ground project components grew by $757 million, or 32 percent, between January 2010 and January 2012. Table 7 identifies growth in estimated contract costs for major program components.

Not only have development contracts experienced rising costs since January 2010, but they have also experienced larger cost increases more recently. For example, between January 2011 and January 2012, contractors’ estimated costs increased by $184 million for the Core Ground System, compared with $88 million for the period between January 2010 and January 2011. Also, between January 2011 and January 2012, contractors’ estimated costs for the spacecraft and the Advanced Baseline Imager increased by $119 million and $91 million compared with $32 million and $57 million, respectively, between January 2010 and January 2011. Figure 4 depicts recent growth in estimated costs for these development components.

The recent growth in contract costs is due in part to the additional labor and engineering support needed to address technical and programmatic problems experienced by flight and ground project components, including the technical complexity associated with development of the Advanced Baseline Imager and the spacecraft, and additional costs associated with the Core Ground System’s revised development plan. NOAA stated that some of the cost growth is attributed to scope changes, including instrument options that were exercised in 2010 and 2011. Based on our analysis, approximately $60 million of the $757 million growth in contractors’ estimated costs at completion from January 2010 through November 2011 was due to scope changes associated with new instrument flight model development. Given the recent increases in contract costs, the program plans to determine how to cover these increased costs by reducing resources applied to other areas of program development and support, delaying scheduled work, or absorbing additional life cycle costs.

Depleted Reserves Could Impact Remaining Development

A contingency reserve (also called management reserve) is important because it provides program managers ready access to funding in order to resolve problems as they occur and may be necessary to cover increased costs resulting from unexpected design complexity, incomplete requirements, or other uncertainties. NOAA requires the program office and flight and ground projects to maintain a reserve of funds until their development is completed. Specifically, the flight project is to maintain 20 percent of planned remaining development costs as reserve, the ground project is to maintain 30 percent of planned remaining development costs as reserve, and the program office is to maintain 10 percent of planned remaining development costs as reserve.

The program has allocated a proportion of its budget as reserves to mitigate risks and manage problems as they surface during development. As a result of changes in budget reserve allocations and reserve commitments, the program’s reserves have declined in recent years. Between January 2009 and January 2012, the program reported that its reserves fell from 42 percent of remaining development costs to 29 percent. Over the same period, the program reports that after accounting for changes in reserve budgets and reserve commitments, reserves fell from $1.7 billion to $1.2 billion, an approximately 30 percent reduction in its uncommitted reserves. This is important to note since about two-thirds of the development remains for the program’s two most expensive components—the spacecraft and the Core Ground System. Recent utilization of the program’s reserves included addressing unanticipated problems with instrument and spacecraft design, using more than expected labor required to accomplish program and project-related milestones, and acquiring additional resources required to execute the revised development plan and schedule for the Core Ground System. In this regard, the program’s independent review board recently raised questions about the sufficiency of the program’s near-term remaining reserves, and program officials decided to establish contractual funding caps for the spacecraft, three instruments—the Advanced Baseline Imager, the Solar Ultraviolet Imager, and the Geostationary Lightning Mapper—and the Core Ground System for fiscal year 2012, delaying work into future years. Figure 5 depicts changes in program reserves over time.

At completion of the GOES-R program’s preliminary design review in February 2012, the program reported that it was within its thresholds because it had maintained 20 percent of planned remaining development costs as reserve for the flight project, 32 percent of planned remaining development costs for the ground project, and 10 percent of planned remaining development costs for the program office. The program will soon enter the integration and test phase, when projects typically experience cost and schedule growth and need additional funding—as identified in our prior reviews of NASA acquisitions. The program may need to draw from its available remaining reserves to address a number of situations, including unanticipated problems during completion of the critical design reviews of the spacecraft and the ground system; unanticipated problems during the integration and test of the instruments and spacecraft, such as required redesign; potential additional labor required for ground system software potential delays in the readiness of NOAA’s Satellite Operations Facility for ground system testing, resulting in the use of contractor facilities; and higher than expected costs for launch vehicles.

While the program reported that reserves were within accepted levels as of February 2012, the reserves may not be matched to remaining development. Although the program restored two satellites to its budget baseline in February 2011, thereby adding approximately $3.2 billion to its total budget, it did not correspondingly change its program reserves. The program did not report its rationale for maintaining reserves at the two-satellite program or explain how these planned reserves were intended to cover risks associated with the development of all four satellites. As a result, there is limited assurance that the reserves are appropriate for each satellite’s remaining development.

Whether the program will continue to stay within its budget depends in part on whether officials have a full understanding of the reserves required for remaining development. Given the program’s recent use of reserves and the significant portions of development remaining for major components, a complete understanding and proper management of reserve levels will be critical to successfully completing all program components. Unless NOAA assesses the reserve allocations across all of the program’s development efforts, it may not be able to ensure that its reserves will cover ongoing challenges as well as unexpected problems for the remaining development of all four satellites in the series.

GOES-R Schedules Are Not Fully Reliable, Contributing to Milestone and Potential Launch Date Delays

The success in management of a large-scale program depends in part on having an integrated and reliable schedule that defines, among other things, when work activities and milestone events will occur, how long they will take, and how they are related to one another. Without such a schedule, program milestones may slip. While the GOES-R program has adopted certain scheduling best practices at both the programwide and contractor levels, unresolved weaknesses also exist, some of which have contributed to current program milestone delays and a replanning of the Core Ground System’s schedule. Without a proper understanding of current program status that a reliable schedule provides, managing the risks of the GOES-R program becomes more difficult and may result in potential delays in GOES-R’s launch date.

GOES-R Integrated Master Schedule and Some Subordinate Schedules Are Unreliable

Program schedules not only provide a road map for systematic program execution, but also provide the means by which to gauge progress, identify and address potential problems, and promote accountability. Accordingly, a schedule helps ensure that all stakeholders understand both the dates for major milestones and the activities that drive the schedule. If changes occur within a program, the schedule helps decision makers analyze how those changes affect the program. The reliability of the schedule will determine the credibility of the program’s forecasted dates, which are used for decision making. Our work has identified nine best practices associated with developing and maintaining a reliable schedule. These are (1) capturing all activities, (2) sequencing all activities, (3) assigning resources to all activities, (4) establishing the duration of all activities, (5) integrating schedule activities horizontally and vertically, (6) establishing the critical path for all activities, (7) identifying reasonable “float” between activities, (8) conducting a schedule risk analysis, and (9) updating the schedule using logic and durations. See table 8 for a description of each of these best practices.

The first seven practices are essential elements for creation of an integrated schedule. An integrated schedule that contains all the detailed tasks necessary to ensure program execution is called an integrated master schedule (IMS). The reliability of an integrated schedule depends in part on the reliability of its subordinate schedules. Automated integration of all activities into a single master schedule can prevent inadvertent errors when entering data or transferring them from one file to another.

GOES-R has an IMS that is created manually once a month directly from at least nine subordinate contractor schedules. Due to anticipated limitations related to the number of activities that would need to be included, program officials stated that they did not intend to make the IMS a live, integrated file. We believe that the lack of a dynamic IMS is an acceptable schedule limitation for an organization of GOES-R’s size and complexity. Program officials also stated that they are in the process of creating an automated process for updating their IMS from contractor- delivered files sometime in 2012, but this was not available in time for our analysis.

To assess the reliability of the programwide IMS, we analyzed four subordinate contractor schedules from July 2011 practices that were met in multiple schedules. For example, similar to the program-level IMS, three of the contractor IMS’s include all activities that were supplemented with monthly data received directly from the schedules of their subcontractors, and the fourth—the Core Ground System schedule—included milestone activities for all subcontractors. All four schedules also substantially or fully met the best practice for regularly updating the schedule, including regular reporting of status and keeping the number of activities completed out of sequence to a minimum. Finally, three of the four schedules substantially met the best practice for establishing accurate activity durations.

Our rationale for the selection and analysis of schedules is discussed in app. I. provide information on schedule risk analyses conducted with risk simulations.

A full set of analysis results is listed in table 9. Selected strengths and weaknesses for each of the four schedules follow the table.

Of the four component schedules, the Geostationary Lightning Mapper schedule demonstrated the most comprehensive implementation of best practices. For instance, all activities and durations were appropriately captured in the schedule; logic links were included for over 99 percent of activities; only one small gap was present in the critical path from the date of the schedule through the end of the project; a high percentage of activities had appropriate logic; and the schedule was updated using logic and durations to determine dates. However, more than 10 percent of the resources listed in the schedule were overloaded, meaning that the schedule required more resources than were available. Also, changes in activity durations for one flight model did not result in a corresponding change in the shipment date of that flight model.

The Advanced Baseline Imager schedule substantially or fully met three of the best practices. As mentioned above, the Advanced Baseline Imager schedule met the best practices for including all subcontractor records as well as regular updating and reporting on the schedule. In addition, it provided a substantial amount of information regarding the resources in its schedule; the contractor created a series of over 900 named codes that denote detailed information such as resource type, location, and labor rate. However, 18 percent of remaining activities had incomplete or missing logic, and 43 percent had soft date constraints. If the schedule is missing logic links between activities, float estimates will not be accurate. Incorrect total float values may in turn result in an invalid critical path and an inaccurate assessment of project completion dates. In the case of the Advanced Baseline Imager, many detailed activities had large total float values throughout. Moreover, the Advanced Baseline Imager schedule did not have a valid critical path for one of its three flight models and did not have a critical path that spans the entire program.

The spacecraft schedule substantially met more than half of the best practices. It had a valid critical path through the launch dates for the first two satellites, included and mapped all activities appropriately, included logic links for over 99 percent of its activities, and was appropriately updated. However, more than 10 percent of its activities had constraints, and nearly 10 percent of its activities had gaps between consecutive activities, also known as lags. Lags should be minimal, and should not be used in place of activities, as they cannot be easily monitored, cannot be included in a risk assessment, and do not take resources. For this reason, lags should be eliminated and replaced with activities so they can be tracked. Also, the spacecraft schedule’s critical path included a year-long current activity for which a detailed breakdown was not available, even though the activity is in the current detailed planning period.

The Core Ground System schedule contained weaknesses across seven of the nine best practice areas resulting in a score of partially or minimally met. For instance, a valid critical path could not be traced between the schedule’s latest status date and the launch date for either of the first two satellites; 12 percent of remaining activities had incomplete or missing logic and 13 percent had soft date constraints; and more than 75 percent of activities had more than 100 days of total float. Also, not all subcontractor detail activities were included in the schedule. Without accounting for all necessary activities, it is uncertain whether all activities are scheduled in the correct order, missing activities would appear on the critical path, or a schedule risk analysis accounts for all risk.

Officials for all four contract teams suggested that certain schedule weaknesses were unique to the July 2011 schedules they provided and that the weaknesses would be remedied in December 2011 schedules. Our subsequent analysis did find improvements for each of the four contractors. For example, the Advanced Baseline Imager’s December schedule had approximately 10 percent fewer activities missing predecessors and successors than in July. The Core Ground System schedule also performed better in three of the nine best practices, including the presence of a full set of duration and total float information, and better information on handoffs with external parties. However, many weaknesses from the July schedules remained in the December schedules. For example, officials stated that several schedule risk analyses had been conducted for the Core Ground System schedule, but also reported that the schedule risk analysis conducted by the Core Ground System’s subcontractor was not valid and did not provide accurate or constructive information.

Of particular importance is the absence of a valid critical path throughout all the schedules. Establishing a valid program-level critical path depends on the resolution of issues with the respective critical paths for the spacecraft and Core Ground System components. Contract specifications for all four contractors require that these schedules define critical paths for their activities. Without a valid critical path, management cannot determine which delayed tasks will have detrimental effects on the project finish date. It also cannot determine if total float within the schedule can be used to mitigate critical tasks by reallocating resources from tasks that can be delayed without launch date impact. Unless the weaknesses in these subordinate schedules are addressed, including the generation of valid critical paths in all schedules, the programwide IMS that is derived from them may not be sufficiently reliable.

Schedule Practice Weaknesses Could Lead to Further Schedule Delays

Weaknesses in implementing scheduling best practices undermine the program’s ability to produce credible dates for planned milestones and events, as illustrated by schedule discrepancies that occurred between the ground project and the flight project and the subsequent replanning that was required. The program has already demonstrated a pattern of milestone delays during its development due in part to the scheduling weaknesses we identified. Although the program has initiated two key efforts that could address certain schedule weaknesses, other weaknesses have not yet been resolved. Until the program’s scheduling weaknesses are corrected, it may experience additional delays to its key milestones.

The program has revised planned milestone and completion dates for each of the instruments as well as the spacecraft and ground system components by at least 3 months—and up to 2 years—since the program originally estimated dates for key milestones in its December 2007 management control plan. Program officials noted that its December 2007 dates were notional estimates until integrated baseline reviews could be conducted. However, delays occurred both before and after the schedules for the instruments, spacecraft, and ground system components were formalized. In certain cases, more recent changes were due to delays in building and testing satellite components. For example, the Solar Ultraviolet Imager experienced delays in 2011 and 2012 due in part to delays in software development and in procuring flight model parts. Also, a failure in testing power supply boards caused rework delays in 2011 and 2012 on the Geostationary Lightning Mapper instrument. See figure 6 for a summary of changes in planned completion dates for components of the flight and ground projects.

The potential for delays remains as GOES-R’s instruments, spacecraft, and ground system components complete their design and testing phases. According to program officials, the Geostationary Lightning Mapper shipment date remains at risk of a potential slip due to redesign efforts that have impacted the release of the build of the electronic board component of the instrument. The current projected delivery for this instrument is August 2013, leaving only 1 month before it is on the critical path for GOES-R’s launch readiness date. Moreover, weaknesses in implementing schedule best practices make meaningful measurement and oversight of program status and progress, as well as accountability for results, difficult to achieve—which can in turn reduce the timeliness and effectiveness of the understanding and mitigation of project risks.

The program office has taken specific positive actions that address two of the scheduling weaknesses we identified. First, the GOES-R program implemented the Giver-Receiver Intersegment Database, a tool that tracks deliverables between the flight and ground projects. Giver-Receiver Intersegment Database items are formally reviewed by various working groups weekly and monthly before they are incorporated into the GOES-R IMS. This initiative is intended to address a program-recognized need for better horizontal integration (related to best practice 5). Second, the GOES- R program implemented a Joint Cost and Schedule Confidence Level, a set of parametric models designed to identify the probability that a given program’s schedule values will be equal to or less than target values on a specific date. In the Joint Cost and Schedule Confidence Level, simulations are run for the expected duration of activities based on probabilities supplied by officials in the project’s cost and schedule division. This initiative is intended to address a program-recognized need to conduct a schedule risk analysis (related to best practice 8).

However, the possibility of future milestone delays remains. Initial results from the Joint Cost and Schedule Confidence Level from January 2012 indicated that there is a 48 percent confidence level that the program will meet its current launch readiness date of October 2015. Program officials plan to consult with the NOAA Program Management Council to determine the advisability of moving the launch readiness date to a 70 percent confidence level of February 2016. Given that scheduling weaknesses remain unaddressed, even these confidence levels may be unreliable. Establishing accurate confidence estimates depends on reliable data that result from the implementation of a full set of scheduling best practices.

Furthermore, delays in GOES-R’s launch date could impact current operational GOES continuity and could produce milestone delays for subsequent satellites in the series. Program documentation indicates that with the current launch readiness date of October 2015, plus an on-orbit testing period, there is a 37 percent chance of a gap in the availability of two operational GOES-series satellites at any one time, assuming a normal lifespan for the satellites currently on-orbit. Any delays in the launch readiness date for GOES-R, which is already at risk due to the increased cost growth and recent heavy use of program reserves discussed previously, would further increase the probability of a gap in satellite continuity. This could result in the need for NOAA to rely on older satellites that are not fully functional. In addition, GOES-R’s schedule reserve is being counted on to complete activities for GOES-S. As a result, delays to certain program schedule targets could impact milestone commitments for GOES-S.

Until the program implements a full set of schedule best practices, and uses it on succeeding schedule updates throughout the life of the program, further delays in the program’s launch dates may occur. In particular, without ensuring that all contractor and subcontractor information is included in the IMS and conducting regular schedule risk assessments, program management may not have timely and relevant information at its disposal for decision making. Lack of the proper understanding of current program status due to schedules that are not fully reliable undercuts the ability of the program office to manage a high- risk program like GOES-R.

Risk Management Approach Is Well Defined but Not Fully Implemented

Risk management is a continuous process to identify potential problems before they occur. By identifying potential problems early, activities can be planned and invoked as needed across the life of a project to avoid or mitigate the adverse impacts of program problems. Effective risk management involves early and aggressive risk identification through the collaboration and involvement of relevant stakeholders. Government and industry risk management guidance divides risk management activities into four key areas—preparing for risk management, identifying and analyzing risks, mitigating risks, and reviewing risks with executive oversight. Table 10 describes recognized best practices in these areas.

NOAA has established policies and procedures for effective risk management for GOES-R. For example, the program has documented a strategy for managing risks that includes important elements, such as relevant stakeholders and their responsibilities and the criteria for evaluating, categorizing, and prioritizing risks. The program’s approved risk management plan also includes requirements for risk mitigation— such as required actions, deadlines, and assigned risk owners—as well as requirements that risks’ status and changes are periodically reviewed by appropriate managers, including senior NOAA and NASA managers. Table 11 compares GOES-R’s risk management policies and procedures with recognized risk management practices for four areas.

With such policies and procedures in place, NOAA has established a comprehensive framework to support its identification, mitigation, and oversight of risks across the program, and laid a foundation for consistent implementation.

The Program Has Not Fully Implemented Its Risk Management Process or Adequately Mitigated Key Risks

While the program has a well-defined risk management process, this process has not been fully implemented. Table 12 identifies the extent to which the program has implemented recognized risk management practices through its risk process. Of particular note, the program has not conducted and documented adequate or timely evaluations for all potential risks—called candidate risks in the risk management plan—in its risk list. In addition, the program did not always document its risk handling strategies and time frames and did not always provide adequate rationale for its decision to close risks.

The GOES-R Program Office has identified 11 risks that it considers critical (medium- or high-level risks) and could significantly impact the program’s development cost or schedule, documented mitigation approaches for each risk, and tracked mitigation progress. For these and its other risks, the program generally has mitigation plans in place that typically include actions to be taken, deadlines, and assigned responsibilities that could help to minimize or control the occurrence of these critical risks. Table 12 describes the program-identified critical risks, as of February 2012.

While the program has documented mitigation plans in place for most of its critical risks, the program is not mitigating its most critical risk— program funding stability. Although the program has included this risk on its top risk list and presented it to the NOAA Program Management Council, it has not devised options for required replanning or functionality descopes should the program experience reduced funding. Program officials stated that the risk is external and beyond their control. However, given that the program has made trade-off decisions regarding available funding, functionality, and the timing of its work, it is reasonable to expect the program to have plans in place that include possible trade-off decisions based on different outcomes, including triggers for when decisions need to be executed.

Further, there is at least one critical risk that is not on the program’s top risk list that could further jeopardize the program’s launch readiness dates and life cycle costs: GOES-S milestones may be affected by delays that have occurred or may still occur during GOES-R development. As discussed earlier in this report, further delays in the development of the first satellite could result in problems for the second satellite’s scheduled milestones since the program is planning to complete the second satellite’s activities during periods of time set aside as reserve to complete the first satellite’s activities. Program officials stated that a risk has recently been added to the flight project’s risk list to reflect the potential delays to the GOES-S development schedule. Until this risk is added to the program’s risk list with a documented mitigation approach and regular monitoring, NOAA could delay the analysis, planning, and actions that would limit the impacts and occurrence of the risk, and would thus be unprepared to face the significant consequences to the program should this risk be realized.

While the program has well-defined policies and procedures, it has not fully implemented its risk process. Fully implementing the recognized risk management practices defined in GOES-R policies and procedures would provide program officials with the assurance that all risks—including those that are new and most critical—are adequately addressed.

Conclusions

NOAA has made progress toward achieving GOES-R program goals by completing certain preliminary and critical design milestones for its flight and ground projects. However, associated reviews, including the programwide critical design review planned for August 2012, have been delayed by many months. Progress was also accompanied by technical and programmatic challenges, such as initial failures of instrument components and replanning of ground project software releases. Although these two specific challenges are being addressed, others such as instrument signal problems and a 2-year cost growth of more than 30 percent for program development contracts point to troubling patterns that will require ongoing remediation and monitoring. NOAA has allocated budget reserves for such situations according to its established guidelines and reports that project reserve levels are currently within those guidelines. However, the program has used approximately 30 percent of its reserves over the last 3 years and significant portions of development remain for major components—including the spacecraft and Core Ground System. In addition, the program did not change its reserves when it restored two satellites adding approximately $3.2 billion to the program’s baseline. Unless NOAA assesses the reserve allocations across all of the program’s development efforts, it may not be able to ensure that its reserves will cover ongoing challenges as well as unexpected problems for the remaining development of all four satellites in the series.

The unreliability of the program’s integrated master schedule and some contractor schedules adds further uncertainty to whether the program will meet its commitments. The issues that exist among these schedules, such as a lack of a full and consistent allocation of resources, incomplete logic, and gaps in the critical path to project completion, are inconsistent with our and agency best practices. NOAA has taken steps to improve schedule reliability through more automated schedule integration, a cross-project deliverable-tracking database, and its first schedule risk assessment. However, unless the program addresses the full set of scheduling weaknesses we identified, its schedules may not provide a fully reliable basis for decision making.

NOAA has defined the policies and procedures it needs to effectively manage and mitigate these and other program risks. Nevertheless, officials do not have a current and comprehensive view of program risk because these policies and procedures have not been fully implemented. Most significantly, not all known candidate risks have been evaluated; corrective actions have not been consistently tracked; and risks we identified in this review are not being tracked or adequately mitigated. Until program officials diligently execute all of the program’s defined risk management practices and integrate these with improvements in the management of reserves and schedules, the program is at risk of exceeding cost and schedule targets and further slipping launch dates for satellites in the GOES-R series.

Recommendations for Executive Action

To improve NOAA’s ability to execute GOES-R’s remaining planned development with appropriate reserves, improve the reliability of its schedules, and address identified program risks, we recommend that the Secretary of Commerce direct the NOAA Administrator to ensure that the following four actions are taken:

Assess and report to the NOAA Program Management Council the reserves needed for completing remaining development for each satellite in the series.

For all satellites in the GOES-R program, including those for which detailed scheduling has yet to begin, address shortfalls in the schedule management practices we identified, including but not limited to incorporating appropriate schedule logic, eliminating unnecessary constraints, creating a realistic allocation of resources, ensuring an unbroken critical path from the current date to the final satellite launch, and ensuring that all subcontractor activities are incorporated in contractors’ integrated master schedules.

Execute the program’s risk management policies and procedures to provide more timely and adequate evaluations and reviews of newly identified risks, documented handling strategies for all ongoing and newly identified risks in the risk register, time frames for when risk mitigation and fallback plans are to be executed, adequate rationale for decisions to close risks, and documentation and tracking of action items from risk review board meetings or other meetings with senior NOAA and NASA managers through completion.

Given the potential impact to the program, add the risk that GOES-S milestones may be affected by GOES-R development to the program’s critical risk list, and ensure that this risk and the program- identified funding stability risk are adequately monitored and mitigated.

Agency Comments and Our Evaluation

We received written comments on a draft of this report from the Secretary of Commerce, who transmitted NOAA’s comments. The department concurred with three of our recommendations and partially concurred with one recommendation. It also provided general comments, which are addressed below, and technical comments, which we have incorporated into our report, as appropriate. A copy of NOAA’s comments is provided in appendix II.

NOAA concurred with our first recommendation to assess and report on the reserves needed for completing the remaining development for each satellite in the series to the agency’s Program Management Council. It stated that the GOES-R program would continue to provide status reports on contingency reserves to the Goddard Space Flight Center and the Program Management Council, and would work with NOAA to ensure contingency reporting meets its needs. It also stated in its general comments that contingency reserve for GOES-T and GOES-U (amounting to 20 percent of their projected development costs) was included when the $3.2 billion for these satellites was added to the program budget baseline in February 2011. NOAA did not provide any additional data on its fiscal year 2012 contingency budget allocations by satellite to support this statement and, as we discussed in this report, the program did not report a significant change in overall program reserve levels when it revised the baseline from two satellites to four satellites.

NOAA concurred with our second recommendation to address shortfalls that we identified in schedule management practices and stated that the program will continue to bring down the number of errors in the schedules and improve the fidelity of the program’s integrated master schedule.

NOAA partially concurred with our third recommendation to fully execute the program’s risk management policies and procedures, to include timely review and disposition of candidate risks. NOAA stated that it did not consider the “concerns” listed in its risk database to be risks or candidate risks, and that the risk management board actively determines whether recorded concerns should be elevated to a risk. However, the program has not treated concerns in accordance with its risk management plan, which considers these to be “candidate risks” and requires their timely review and disposition, as evidenced by the many concerns in the database that were more than 3 months old and had not been assessed or dispositioned. Unless NOAA follows its risk management plan by promptly evaluating “concerns,” it cannot ensure that it is adequately managing the full set of risks that could impact the program.

NOAA concurred with our fourth recommendation to add to the program’s critical risk list that GOES-S milestones may be affected by GOES-R development and to ensure that this risk and the program-identified funding stability risk are adequately monitored and mitigated. NOAA stated that it currently has identified and reported on the GOES-S milestone risk and that the program is working with NOAA to monitor and mitigate the funding stability risk.

In a general comment on our discussion of contractors’ estimated cost growth, NOAA stated that certain growth in contract costs was due to scope changes, including instrument options that were exercised in 2010 and 2011, and stated that such changes should be distinguished from cost growth due to efforts to resolve problems. We consider scope changes to be a valid component of program cost growth. Nevertheless, we revised our report to state that approximately $60 million of the $757 million growth in contractors’ estimated costs at completion from January 2010 through November 2011 was due to scope changes associated with new instrument flight model development. If these scope changes are excluded, contractors’ estimated costs for flight and ground project components still grew by approximately $700 million, or approximately 30 percent, between January 2010 and January 2012.

NOAA commented that the GOES-R launch readiness date changed from December 2014 to October 2015 due to a protest against the spacecraft contract award. We acknowledged the reasons for delays in GOES-R’s launch readiness date in our draft report submitted to NOAA and in our prior reports on this program, and have added language to this report that relates to the bid protest and NASA’s response in delaying the award.

NOAA also commented that the revised incremental schedule was less risky than its original waterfall schedule. While we acknowledged the reduced risk in our draft report and have further revised our report to reflect NOAA’s comments on our draft, we also note that the revised development methodology introduced other risks to the program—such as additional contractor staff and software development and verification activities that require government oversight and continuous monitoring.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 1 day from the report date. At that time, we will send copies to interested congressional committees, the Secretary of Commerce, the Administrator of NASA, the Director of the Office of Management and Budget, and other interested parties. The report also will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staffs have any questions on the matters discussed in this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) assess the National Oceanic and Atmospheric Administration’s (NOAA) progress in developing the Geostationary Operational Environmental Satellite-R series (GOES-R) program, (2) evaluate whether the agency has a reliable schedule for executing the program, and (3) determine whether the program is applying best practices in managing and mitigating its risks.

To assess progress in developing the GOES-R satellite program, we compared the program’s planned completion dates for key milestones identified in its management control plan and system review plan against actual and current estimated completion dates of milestones. We analyzed program monthly status briefings to identify the current status and recent development challenges of flight and ground project components and instruments. We also analyzed contractor- and program- reported data of development costs and reserves. Finally, we conducted several interviews with GOES-R program staff to better understand milestone time frames, to discuss current status and recent development challenges for work currently being performed on GOES-R, and to understand how the program reports costs and reserve totals.

To evaluate whether NOAA has a reliable schedule for executing the program, we evaluated contractor schedules to determine the extent to which GOES-R is following our identified best practices in creating and maintaining its schedules. We analyzed four contractor schedules—two of which represent the overall schedules for the flight and ground projects (spacecraft and Core Ground System), the program’s critical flight instrument (Advanced Baseline Imager), and an instrument that had been experiencing implementation issues at the time of our review (Geostationary Lightning Mapper). We also populated workbooks as a part of that analysis to highlight potential areas of strengths and weakness in schedule logic, use of resources, task duration, float, and task completion; analyzed programwide initiatives undertaken by GOES- R such as the Joint Cost and Schedule Confidence Level and the Giver- Receiver Intersegment Database; assessed GOES-R’s progress against their own scheduling requirements, and interviewed government and contractor officials regarding their scheduling practices.

To determine whether the GOES-R program is applying best practices in managing and mitigating its risks, we analyzed the program’s risk management plan to identify the program’s policies and procedures and compared these to outputs from the program’s risk register and other program risk management documentation, such as risk mitigation status reports. We also reviewed documents such as monthly briefings to the NOAA Program Management Council and program risk status reports to determine how individual risks are managed and reviewed on a regular basis. We assessed the extent to which the program’s policies and procedures met government and industry recognized risk management practices from the Software Engineering Institute’s Capability Maturity Model® Integration for Acquisition (Version 1.3), the International Organization for Standardization, the Defense Acquisition University, and GAO, and whether the program fully implemented its policies and procedures. We also examined the program’s risk status reports and risk register to identify risks for which the program did not have an adequate mitigation plan or were not being actively monitored by the program. Finally, we interviewed GOES-R officials to gain further insight into the program’s risk management process and the program’s application of its process.

We primarily performed our work at NOAA and National Aeronautics and Space Administration offices in the Washington, D.C., metropolitan area. We conducted this performance audit from August 2011 to June 2012, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Commerce

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, individuals making contributions to this report included Colleen Phillips (assistant director), Paula Moore (assistant director), Shaun Byrnes, Nancy Glover, Franklin Jackson, Jason Lee, and Josh Leiling.