Geostationary Weather Satellites: Design Progress Made, but Schedule Uncertainty Needs to be Addressed
Published: 2012-06-26T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The GOES-R series is a set of four satellites intended to replace existing weather satellites that will likely reach the end of their useful lives in about 2015. NOAA estimates the series to cost $10.9 billion through 2036. Because the transition to the series is critical to the nations ability to maintain the continuity of data required for weather forecasting, GAO reviewed NOAAs management of the GOES-R program.
Specifically, GAO was asked to (1) assess NOAAs progress in developing the GOES-R satellite program, (2) evaluate whether the agency has a reliable schedule for executing the program, and (3) determine whether the program is applying best practices in managing and mitigating its risks.
GAO analyzed program management, acquisition, and cost data; evaluated contractor and program-wide schedules against best practices; analyzed program documentation including risk management plans and procedures; and interviewed government and contractor staff regarding program progress and challenges.
section_title: What GAO Found
paragraphs: The Geostationary Operational Environmental Satellite-R series (GOES-R) program has made progress by completing its early design milestones and is nearing the end of the design phase for its spacecraft, instrument, and ground system components. While the program continues to make progress, recent technical problems with the instruments and spacecraft, as well as a significant modification to the ground projects development plan, have delayed the completion of key reviews and led to increased complexity for the development of GOES-R. The technical and programmatic challenges experienced by the flight and ground projects have led to a 19-month delay in completing the programs preliminary design review. Nevertheless, program officials report that its planned launch date of October 2015 for the first satellite has not changed. While the program reports that approximately $1.2 billion is currently in reserve to manage future delays and cost growth, significant portions of development remain for major components. As a result, the program may not be able to ensure that it has adequate resources to cover ongoing challenges as well as unexpected problems for the remaining development of all four satellites.
The success in management of a large-scale program depends in part on having a reliable schedule that defines, among other things, when work activities and milestone events will occur, how long they will take, and how they are related to one another. To its credit, the program has adopted key scheduling best practices and has recognized certain scheduling weaknesses. It has also recently instituted initiatives to automate its integrated master schedule, correct integration problems among projects, and assess schedule confidence based on risk. However, unresolved schedule deficiencies remain in its integrated master schedule and the contractor schedules that support it, which have contributed to a re-plan of the schedule of the ground system and to the potential for delays to satellite launch dates. The program recently determined that the likelihood of the first satellite meeting its planned October 2015 launch date is 48 percent. Based on this planned launch date, the program reports that there is a 37 percent chance of a gap in the availability of two operational GOES-series satellites, which could result in the need for the National Oceanic and Atmospheric Administration (NOAA) to rely on older satellites that are not fully functional. Until its scheduling weaknesses are addressed, it will be more difficult for the program to know whether its planned remaining development is on schedule.
NOAA has established policies and procedures that conform with recognized risk management best practices. For example, the program has documented a strategy for managing risks that includes important elements, such as relevant stakeholders and their responsibilities and the criteria for evaluating, categorizing, and prioritizing risks. However, while the program has a well- defined risk management process, it has not been fully implemented. For example, the program has not provided adequate or timely evaluations for potential risks, did not always provide adequate rationale for the decision to close a risk, and has at least two critical risks in need of additional attention. Until all defined risk management practices are diligently executed and critical risks adequately mitigated, the GOES-R program is at risk of exceeding cost and schedule targets, and launch dates could slip.
section_title: What GAO Recommends
paragraphs: GAO is making recommendations to NOAA to assess and report reserves needed over the life of the program, improve the reliability of its schedules, and address identified program risks. NOAA concurred or partially concurred with GAOs recommendations.

section_title: Letter
paragraphs: Operational geostationary environmental satellites play a critical role in  our nation’s weather forecasting. These satellites—which are managed  by the Department of Commerce’s National Oceanic and Atmospheric  Administration (NOAA)—provide critical information on atmospheric,  oceanic, climatic, and solar conditions that help meteorologists observe  and predict global and local weather events. They also provide a means  of identifying the large-scale evolution of severe storms, such as  hurricane track and intensity forecasting and complementing radar in  tornado or heavy precipitation forecasting.
NOAA, through collaboration with the National Aeronautics and Space  Administration (NASA), is procuring the next generation of geostationary  satellites, called the Geostationary Operational Environmental Satellite-R  (GOES-R) series. The GOES-R series is to replace the current series of  satellites, which will likely begin to reach the end of their useful lives in  approximately 2015. This new series is expected to mark the first major  improvement in GOES instrumentation since 1994. It is also considered  critical to the United States’ ability to maintain the continuity of data  required for weather forecasting through the year 2036.
This report responds to your request that we review NOAA’s  management of the GOES-R program. Specifically, we were asked to (1)  assess NOAA’s progress in developing the GOES-R satellite program, (2)  evaluate whether the agency has a reliable schedule for executing the  program, and (3) determine whether the program is applying best  practices in managing and mitigating its risks.
To assess NOAA’s progress in developing GOES-R, we analyzed  program documentation, including monthly status reports, acquisition  plans, and contractor performance reports on development efforts. To  evaluate whether the agency has a reliable schedule for executing the  program, we evaluated four contractor schedules including two  instruments—the Advanced Baseline Imager and Geostationary Lightning  Mapper—the spacecraft, and the Core Ground System. We also analyzed  programwide schedule initiatives. We compared contractor schedules and  program initiatives to GAO-identified best practices in developing and  maintaining schedules. To determine whether the program is applying  best practices in managing and mitigating its risks, we analyzed program  documentation, including the program’s risk management plan and  outputs from its risk register. We compared program risk management  policies and procedures to government and industry recognized risk  management best practices and compared the program’s implementation  of its risk management policies and procedures. We also interviewed  government and contractor staff to discuss the program’s development  progress and recent challenges as well as their scheduling and risk  management practices.
We conducted this performance audit from August 2011 to June 2012 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Appendix I contains further  details on our objectives, scope, and methodology.
subsections: section_title: Background
paragraphs: Since the 1960s, geostationary satellites have been used by the United  States to provide meteorological data for weather observation, research,  and forecasting. NOAA’s National Environmental Satellite, Data, and  Information Service is responsible for managing the civilian operational  geostationary satellite system, called GOES. Geostationary satellites can  maintain a constant view of the earth from a high orbit of about 22,300  miles in space.
NOAA operates GOES as a two-satellite system that is primarily focused  on the United States (see fig. 1). These satellites provide timely  environmental data about the earth’s atmosphere, surface, cloud cover,  and the space environment to meteorologists and their audiences. They  also observe the development of hazardous weather, such as hurricanes  and severe thunderstorms, and track their movement and intensity to  reduce or avoid major losses of property and life. The ability of the  satellites to provide broad, continuously updated coverage of atmospheric  conditions over land and oceans is important to NOAA’s weather  forecasting operations.
To provide continuous satellite coverage, NOAA acquires several  satellites at a time as part of a series and launches new satellites every  few years (see table 1).satellites and one backup satellite in orbit at all times.
section_title: Progress Made in Completing Early Design, but Milestones Were Completed Late and Development Costs Have Increased
paragraphs: While the GOES-R program has made progress in completing its early  design and is nearing the end of the design phase for its flight and ground  system components, it completed key design milestones later than  planned. Recent technical problems with the instruments and spacecraft,  as well as a significant modification to the ground project’s development  plan, have delayed the completion of key reviews and led to increased  complexity for GOES-R’s development. Moreover, several instrument,  spacecraft, and ground system problems identified during design, have  not yet been resolved. In addition to the delays, the technical and  programmatic challenges experienced by GOES-R’s flight and ground  projects have led to increased costs for its development contracts.  Despite these problems, program officials report that planned launch  dates and cost estimates for the first two satellites have not changed, and  that approximately $1.2 billion is currently in reserve to manage future  delays and cost growth. However, the program has used approximately  30 percent of its reserves over the last 3 years and significant portions of  development remain for major components—including the spacecraft and  Core Ground System. In addition, the program did not change its  reserves when it restored two satellites adding approximately $3.2 billion  to the program’s baseline. As a result, the program will be challenged in  completing its remaining development, particularly the final design and  testing of the spacecraft and ground system, within its cost and schedule  targets.
subsections: section_title: GOES-R Has Made Progress on Its Design, but Initial Design Milestones Completed Later than Planned
paragraphs: Two key types of development milestones, which are identified in the  GOES-R December 2007 management control plan, are the preliminary  design review (PDR) and a more detailed critical design review (CDR).  The PDR is an initial design milestone that assesses the readiness of the  program to proceed with detailed design activities and the CDR is  intended to demonstrate that the design is complete and appropriate to  support proceeding to full-scale software development, as well as  fabrication, assembly, integration, and testing. The program planned to  complete the flight project’s PDR in April 2010 and CDR in April 2011. It  planned to complete the ground project’s PDR in July 2010 and CDR in  July 2011. These project-level reviews are to be completed before the  program’s comprehensive PDR and CDR can be completed.
The program has demonstrated progress toward completing its design.  Specifically, the program and its subsequent projects have successfully  completed their PDRs, demonstrating that they are ready to proceed with  detailed design activities. The program and its projects are also currently  progressing towards the final design for the entire GOES-R system, which  is expected to be completed at the program’s CDR planned for   August 2012.
However, the program’s PDR milestones were completed later than their  2007 planned dates. Consequently, its CDR milestones were similarly  delayed. Table 4 highlights delays in key program milestones as of   April 2012.
Going forward, assembly and testing of all flight instruments for the first  satellite in the series is to be completed by August 2013. Both the flight  and ground projects’ development components are expected to be  complete by September 2015. Despite recent delays in program  milestones, NOAA still expects to meet an October 2015 launch date for  the first satellite in the series by utilizing planned schedule reserves.
The Flight Project Office has made progress by completing the critical  design reviews for each of its five main instruments, which is significant  because the instrument designs will be applied to each satellite in the  GOES-R series. The Geostationary Lightning Mapper was the most  recent instrument to complete its critical design review, which occurred in  August 2011—16 months after its planned completion date. Instrument  fabrication, assembly, and testing activities are under way. The final  instrument scheduled for completion—the Geostationary Lightning  Mapper—is expected to be delivered for integration with the spacecraft by  August 2013.
Although instrument design milestones are complete and assembly and  testing activities are under way, each of the instruments and the  spacecraft has recently encountered technical challenges. The Flight  Project Office has taken steps to resolve several technical problems  through additional engineering support and redesign efforts. However,  there are still important technical challenges to be addressed, including  signal blurring problems for several of the Advanced Baseline Imager’s  infrared channels and Geostationary Lightning Mapper emissions that are  exceeding specifications. The project office is monitoring these problems  and has plans in place to address them. The current development efforts  and recent challenges for components of the flight project are described  in table 5.
The Ground Project Office has made progress in completing the ground  system’s preliminary design. However, in doing so it experienced  problems in defining ground system software requirements and identified  problems with its dependencies on flight project schedules. Specifically,  the project office discovered in early 2011 that software design  requirements had not progressed enough to conduct the ground system’s  preliminary design review. In addition, the ground system’s development  schedule included software deliveries from flight project instruments that  were not properly integrated—they had not yet been defined or could not  be met. To address these problems, the Ground Project Office made  significant revisions to the Core Ground System’s baseline development  plan and schedule.
In order to avoid potential slippages to GOES-R’s launch date, project  officials decided to switch from plans to deliver software capabilities at  major software releases to an approach where software capabilities could  be delivered incrementally (prior to major releases) as the project  received data inputs from the flight project. According to NOAA and  program officials, the revised development approach is to provide  flexibility in the ground system’s development schedule and reduce risk  associated with the original waterfall schedule. However, the revised plan  is expected to cost $85 million more than the original plan through the  Core Ground System’s completion. This cost includes increased  contractor and government staff, new oversight tools, and more  verification and testing activities associated with an increased number of  software deliveries. Program officials acknowledged that the revised plan  intends to expend additional resources to reduce schedule risk and  potential impacts to GOES-R’s launch date, but that it also introduces  new cost and schedule risks associated with incremental development,  such as more software development and verification activities that require  additional government oversight and continuous monitoring.
In addition, the program has cancelled previously exercised options to the  ground project that were once considered part of its original baseline. In  early 2011, the program determined that it could no longer fund Core  Ground System contract options—which were estimated to cost  approximately $50 million. Program officials stated that they cancelled the  contract options due to approaching development commitments, including  revisions to the ground system’s revised plan and schedule, and funding  reductions from fiscal year 2011. According to program officials, the work  to be performed under the cancelled contract options could be addressed  by NOAA after GOES-R satellites are launched; however, there are  currently no plans in place to do so.
Table 6 describes the development efforts and recent challenges for  ground project components.
The GOES-R program’s estimated costs include, among other things,  actual and estimated contract costs associated with design, development,  integration, and testing activities for the instruments, spacecraft, and  ground system as well as procurement of the satellites’ launch vehicles.  The estimated costs also include government costs such as NOAA and  NASA program management support and contingency reserves to be  applied towards critical risks and issues as they arise. As of January  2010, the program reported estimates of $3.3 billion for the flight project  (including reserves of $598 million), $1.7 billion for the ground project  (including reserves of $431 million), $2.0 billion for other program costs  (including reserves of $617 million), and $748 million for operations and  maintenance.
Although NOAA has not changed its program cost estimates for the  development of GOES-R and GOES-S, contract costs for the  instruments, spacecraft, and ground system are rising. Specifically,  contractor estimated costs for flight and ground project components grew  by $757 million, or 32 percent, between January 2010 and January 2012.  Table 7 identifies growth in estimated contract costs for major program  components.
Not only have development contracts experienced rising costs since  January 2010, but they have also experienced larger cost increases more  recently. For example, between January 2011 and January 2012,  contractors’ estimated costs increased by $184 million for the Core  Ground System, compared with $88 million for the period between  January 2010 and January 2011. Also, between January 2011 and  January 2012, contractors’ estimated costs for the spacecraft and the  Advanced Baseline Imager increased by $119 million and $91 million  compared with $32 million and $57 million, respectively, between January  2010 and January 2011. Figure 4 depicts recent growth in estimated  costs for these development components.
The recent growth in contract costs is due in part to the additional labor  and engineering support needed to address technical and programmatic  problems experienced by flight and ground project components, including  the technical complexity associated with development of the Advanced  Baseline Imager and the spacecraft, and additional costs associated with  the Core Ground System’s revised development plan. NOAA stated that  some of the cost growth is attributed to scope changes, including  instrument options that were exercised in 2010 and 2011. Based on our  analysis, approximately $60 million of the $757 million growth in  contractors’ estimated costs at completion from January 2010 through  November 2011 was due to scope changes associated with new  instrument flight model development. Given the recent increases in  contract costs, the program plans to determine how to cover these  increased costs by reducing resources applied to other areas of program  development and support, delaying scheduled work, or absorbing  additional life cycle costs.
section_title: Depleted Reserves Could Impact Remaining Development
paragraphs: A contingency reserve (also called management reserve) is important  because it provides program managers ready access to funding in order  to resolve problems as they occur and may be necessary to cover  increased costs resulting from unexpected design complexity, incomplete  requirements, or other uncertainties. NOAA requires the program office  and flight and ground projects to maintain a reserve of funds until their  development is completed. Specifically, the flight project is to maintain 20  percent of planned remaining development costs as reserve, the ground  project is to maintain 30 percent of planned remaining development costs  as reserve, and the program office is to maintain 10 percent of planned  remaining development costs as reserve.
The program has allocated a proportion of its budget as reserves to  mitigate risks and manage problems as they surface during development.  As a result of changes in budget reserve allocations and reserve  commitments, the program’s reserves have declined in recent years.  Between January 2009 and January 2012, the program reported that its  reserves fell from 42 percent of remaining development costs to 29  percent. Over the same period, the program reports that after accounting  for changes in reserve budgets and reserve commitments, reserves fell  from $1.7 billion to $1.2 billion, an approximately 30 percent reduction in its  uncommitted reserves. This is important to note since about two-thirds of  the development remains for the program’s two most expensive  components—the spacecraft and the Core Ground System. Recent  utilization of the program’s reserves included addressing unanticipated  problems with instrument and spacecraft design, using more than expected  labor required to accomplish program and project-related milestones, and  acquiring additional resources required to execute the revised development  plan and schedule for the Core Ground System. In this regard, the  program’s independent review board recently raised questions about the  sufficiency of the program’s near-term remaining reserves, and program  officials decided to establish contractual funding caps for the spacecraft,  three instruments—the Advanced Baseline Imager, the Solar Ultraviolet  Imager, and the Geostationary Lightning Mapper—and the Core Ground  System for fiscal year 2012, delaying work into future years. Figure 5  depicts changes in program reserves over time.
At completion of the GOES-R program’s preliminary design review in  February 2012, the program reported that it was within its thresholds  because it had maintained 20 percent of planned remaining development  costs as reserve for the flight project, 32 percent of planned remaining  development costs for the ground project, and 10 percent of planned  remaining development costs for the program office. The program will  soon enter the integration and test phase, when projects typically  experience cost and schedule growth and need additional funding—as  identified in our prior reviews of NASA acquisitions. The program may  need to draw from its available remaining reserves to address a number  of situations, including unanticipated problems during completion of the critical design  reviews of the spacecraft and the ground system; unanticipated problems during the integration and test of the  instruments and spacecraft, such as required redesign; potential additional labor required for ground system software potential delays in the readiness of NOAA’s Satellite Operations  Facility for ground system testing, resulting in the use of contractor  facilities; and higher than expected costs for launch vehicles.
While the program reported that reserves were within accepted levels as  of February 2012, the reserves may not be matched to remaining  development. Although the program restored two satellites to its budget  baseline in February 2011, thereby adding approximately $3.2 billion to its  total budget, it did not correspondingly change its program reserves.  The program did not report its rationale for maintaining reserves at the  two-satellite program or explain how these planned reserves were  intended to cover risks associated with the development of all four  satellites. As a result, there is limited assurance that the reserves are  appropriate for each satellite’s remaining development.
Whether the program will continue to stay within its budget depends in  part on whether officials have a full understanding of the reserves  required for remaining development. Given the program’s recent use of  reserves and the significant portions of development remaining for major  components, a complete understanding and proper management of  reserve levels will be critical to successfully completing all program  components. Unless NOAA assesses the reserve allocations across all of  the program’s development efforts, it may not be able to ensure that its  reserves will cover ongoing challenges as well as unexpected problems  for the remaining development of all four satellites in the series.
section_title: GOES-R Schedules Are Not Fully Reliable, Contributing to Milestone and Potential Launch Date Delays
paragraphs: The success in management of a large-scale program depends in part on  having an integrated and reliable schedule that defines, among other  things, when work activities and milestone events will occur, how long  they will take, and how they are related to one another. Without such a  schedule, program milestones may slip. While the GOES-R program has  adopted certain scheduling best practices at both the programwide and  contractor levels, unresolved weaknesses also exist, some of which have  contributed to current program milestone delays and a replanning of the  Core Ground System’s schedule. Without a proper understanding of  current program status that a reliable schedule provides, managing the  risks of the GOES-R program becomes more difficult and may result in  potential delays in GOES-R’s launch date.
subsections: section_title: GOES-R Integrated Master Schedule and Some Subordinate Schedules Are Unreliable
paragraphs: Program schedules not only provide a road map for systematic program  execution, but also provide the means by which to gauge progress,  identify and address potential problems, and promote accountability.  Accordingly, a schedule helps ensure that all stakeholders understand  both the dates for major milestones and the activities that drive the  schedule. If changes occur within a program, the schedule helps decision  makers analyze how those changes affect the program. The reliability of  the schedule will determine the credibility of the program’s forecasted  dates, which are used for decision making. Our work has identified nine  best practices associated with developing and maintaining a reliable  schedule. These are (1) capturing all activities, (2) sequencing all  activities, (3) assigning resources to all activities, (4) establishing the  duration of all activities, (5) integrating schedule activities horizontally and  vertically, (6) establishing the critical path for all activities, (7) identifying  reasonable “float” between activities, (8) conducting a schedule risk  analysis, and (9) updating the schedule using logic and durations. See  table 8 for a description of each of these best practices.
The first seven practices are essential elements for creation of an  integrated schedule. An integrated schedule that contains all the detailed  tasks necessary to ensure program execution is called an integrated  master schedule (IMS). The reliability of an integrated schedule depends  in part on the reliability of its subordinate schedules. Automated  integration of all activities into a single master schedule can prevent  inadvertent errors when entering data or transferring them from one file to  another.
GOES-R has an IMS that is created manually once a month directly from  at least nine subordinate contractor schedules. Due to anticipated  limitations related to the number of activities that would need to be  included, program officials stated that they did not intend to make the IMS  a live, integrated file. We believe that the lack of a dynamic IMS is an  acceptable schedule limitation for an organization of GOES-R’s size and  complexity. Program officials also stated that they are in the process of  creating an automated process for updating their IMS from contractor- delivered files sometime in 2012, but this was not available in time for our  analysis.
To assess the reliability of the programwide IMS, we analyzed four  subordinate contractor schedules from July 2011 practices that were met in multiple schedules. For example, similar to the  program-level IMS, three of the contractor IMS’s include all activities that  were supplemented with monthly data received directly from the  schedules of their subcontractors, and the fourth—the Core Ground  System schedule—included milestone activities for all subcontractors. All  four schedules also substantially or fully met the best practice for  regularly updating the schedule, including regular reporting of status and  keeping the number of activities completed out of sequence to a  minimum. Finally, three of the four schedules substantially met the best  practice for establishing accurate activity durations.
Our rationale for the selection and analysis of schedules is discussed in app. I. provide information on schedule risk analyses conducted with risk  simulations.
A full set of analysis results is listed in table 9. Selected strengths and  weaknesses for each of the four schedules follow the table.
Of the four component schedules, the Geostationary Lightning Mapper  schedule demonstrated the most comprehensive implementation of best  practices. For instance, all activities and durations were appropriately  captured in the schedule; logic links were included for over 99 percent of  activities; only one small gap was present in the critical path from the date  of the schedule through the end of the project; a high percentage of  activities had appropriate logic; and the schedule was updated using logic  and durations to determine dates. However, more than 10 percent of the  resources listed in the schedule were overloaded, meaning that the  schedule required more resources than were available. Also, changes in  activity durations for one flight model did not result in a corresponding  change in the shipment date of that flight model.
The Advanced Baseline Imager schedule substantially or fully met three  of the best practices. As mentioned above, the Advanced Baseline  Imager schedule met the best practices for including all subcontractor  records as well as regular updating and reporting on the schedule. In  addition, it provided a substantial amount of information regarding the  resources in its schedule; the contractor created a series of over 900  named codes that denote detailed information such as resource type,  location, and labor rate. However, 18 percent of remaining activities had  incomplete or missing logic, and 43 percent had soft date constraints. If  the schedule is missing logic links between activities, float estimates will  not be accurate. Incorrect total float values may in turn result in an invalid  critical path and an inaccurate assessment of project completion dates. In  the case of the Advanced Baseline Imager, many detailed activities had  large total float values throughout. Moreover, the Advanced Baseline  Imager schedule did not have a valid critical path for one of its three flight  models and did not have a critical path that spans the entire program.
The spacecraft schedule substantially met more than half of the best  practices. It had a valid critical path through the launch dates for the first  two satellites, included and mapped all activities appropriately, included  logic links for over 99 percent of its activities, and was appropriately  updated. However, more than 10 percent of its activities had constraints,  and nearly 10 percent of its activities had gaps between consecutive  activities, also known as lags. Lags should be minimal, and should not be  used in place of activities, as they cannot be easily monitored, cannot be  included in a risk assessment, and do not take resources. For this reason,  lags should be eliminated and replaced with activities so they can be  tracked. Also, the spacecraft schedule’s critical path included a year-long  current activity for which a detailed breakdown was not available, even  though the activity is in the current detailed planning period.
The Core Ground System schedule contained weaknesses across seven  of the nine best practice areas resulting in a score of partially or minimally  met. For instance, a valid critical path could not be traced between the  schedule’s latest status date and the launch date for either of the first two  satellites; 12 percent of remaining activities had incomplete or missing  logic and 13 percent had soft date constraints; and more than 75 percent  of activities had more than 100 days of total float. Also, not all  subcontractor detail activities were included in the schedule. Without  accounting for all necessary activities, it is uncertain whether all activities  are scheduled in the correct order, missing activities would appear on the  critical path, or a schedule risk analysis accounts for all risk.
Officials for all four contract teams suggested that certain schedule  weaknesses were unique to the July 2011 schedules they provided and  that the weaknesses would be remedied in December 2011 schedules.  Our subsequent analysis did find improvements for each of the four  contractors. For example, the Advanced Baseline Imager’s December  schedule had approximately 10 percent fewer activities missing  predecessors and successors than in July. The Core Ground System  schedule also performed better in three of the nine best practices,  including the presence of a full set of duration and total float information,  and better information on handoffs with external parties. However, many  weaknesses from the July schedules remained in the December  schedules. For example, officials stated that several schedule risk  analyses had been conducted for the Core Ground System schedule, but  also reported that the schedule risk analysis conducted by the Core  Ground System’s subcontractor was not valid and did not provide  accurate or constructive information.
Of particular importance is the absence of a valid critical path throughout  all the schedules. Establishing a valid program-level critical path depends  on the resolution of issues with the respective critical paths for the  spacecraft and Core Ground System components. Contract specifications  for all four contractors require that these schedules define critical paths  for their activities. Without a valid critical path, management cannot  determine which delayed tasks will have detrimental effects on the project  finish date. It also cannot determine if total float within the schedule can  be used to mitigate critical tasks by reallocating resources from tasks that  can be delayed without launch date impact. Unless the weaknesses in  these subordinate schedules are addressed, including the generation of  valid critical paths in all schedules, the programwide IMS that is derived  from them may not be sufficiently reliable.
section_title: Schedule Practice Weaknesses Could Lead to Further Schedule Delays
paragraphs: Weaknesses in implementing scheduling best practices undermine the  program’s ability to produce credible dates for planned milestones and  events, as illustrated by schedule discrepancies that occurred between  the ground project and the flight project and the subsequent replanning  that was required. The program has already demonstrated a pattern of  milestone delays during its development due in part to the scheduling  weaknesses we identified. Although the program has initiated two key  efforts that could address certain schedule weaknesses, other  weaknesses have not yet been resolved. Until the program’s scheduling  weaknesses are corrected, it may experience additional delays to its key  milestones.
The program has revised planned milestone and completion dates for  each of the instruments as well as the spacecraft and ground system  components by at least 3 months—and up to 2 years—since the program  originally estimated dates for key milestones in its December 2007  management control plan. Program officials noted that its December 2007  dates were notional estimates until integrated baseline reviews could be  conducted. However, delays occurred both before and after the  schedules for the instruments, spacecraft, and ground system  components were formalized. In certain cases, more recent changes  were due to delays in building and testing satellite components. For  example, the Solar Ultraviolet Imager experienced delays in 2011 and  2012 due in part to delays in software development and in procuring flight  model parts. Also, a failure in testing power supply boards caused rework  delays in 2011 and 2012 on the Geostationary Lightning Mapper  instrument. See figure 6 for a summary of changes in planned completion  dates for components of the flight and ground projects.
The potential for delays remains as GOES-R’s instruments, spacecraft,  and ground system components complete their design and testing  phases. According to program officials, the Geostationary Lightning  Mapper shipment date remains at risk of a potential slip due to redesign  efforts that have impacted the release of the build of the electronic board  component of the instrument. The current projected delivery for this  instrument is August 2013, leaving only 1 month before it is on the critical  path for GOES-R’s launch readiness date. Moreover, weaknesses in  implementing schedule best practices make meaningful measurement  and oversight of program status and progress, as well as accountability  for results, difficult to achieve—which can in turn reduce the timeliness  and effectiveness of the understanding and mitigation of project risks.
The program office has taken specific positive actions that address two of  the scheduling weaknesses we identified. First, the GOES-R program  implemented the Giver-Receiver Intersegment Database, a tool that tracks  deliverables between the flight and ground projects. Giver-Receiver  Intersegment Database items are formally reviewed by various working  groups weekly and monthly before they are incorporated into the GOES-R  IMS. This initiative is intended to address a program-recognized need for  better horizontal integration (related to best practice 5). Second, the GOES- R program implemented a Joint Cost and Schedule Confidence Level, a  set of parametric models designed to identify the probability that a given  program’s schedule values will be equal to or less than target values on a  specific date. In the Joint Cost and Schedule Confidence Level,  simulations are run for the expected duration of activities based on  probabilities supplied by officials in the project’s cost and schedule division.  This initiative is intended to address a program-recognized need to conduct  a schedule risk analysis (related to best practice 8).
However, the possibility of future milestone delays remains. Initial results  from the Joint Cost and Schedule Confidence Level from January 2012  indicated that there is a 48 percent confidence level that the program will  meet its current launch readiness date of October 2015. Program officials  plan to consult with the NOAA Program Management Council to  determine the advisability of moving the launch readiness date to a 70  percent confidence level of February 2016. Given that scheduling  weaknesses remain unaddressed, even these confidence levels may be  unreliable. Establishing accurate confidence estimates depends on  reliable data that result from the implementation of a full set of scheduling  best practices.
Furthermore, delays in GOES-R’s launch date could impact current  operational GOES continuity and could produce milestone delays for  subsequent satellites in the series. Program documentation indicates that  with the current launch readiness date of October 2015, plus an on-orbit  testing period, there is a 37 percent chance of a gap in the availability of  two operational GOES-series satellites at any one time, assuming a  normal lifespan for the satellites currently on-orbit. Any delays in the  launch readiness date for GOES-R, which is already at risk due to the  increased cost growth and recent heavy use of program reserves  discussed previously, would further increase the probability of a gap in  satellite continuity. This could result in the need for NOAA to rely on older  satellites that are not fully functional. In addition, GOES-R’s schedule  reserve is being counted on to complete activities for GOES-S. As a  result, delays to certain program schedule targets could impact milestone  commitments for GOES-S.
Until the program implements a full set of schedule best practices, and  uses it on succeeding schedule updates throughout the life of the  program, further delays in the program’s launch dates may occur. In  particular, without ensuring that all contractor and subcontractor  information is included in the IMS and conducting regular schedule risk  assessments, program management may not have timely and relevant  information at its disposal for decision making. Lack of the proper  understanding of current program status due to schedules that are not  fully reliable undercuts the ability of the program office to manage a high- risk program like GOES-R.
section_title: Risk Management Approach Is Well Defined but Not Fully Implemented
paragraphs: Risk management is a continuous process to identify potential problems  before they occur. By identifying potential problems early, activities can  be planned and invoked as needed across the life of a project to avoid or  mitigate the adverse impacts of program problems. Effective risk  management involves early and aggressive risk identification through the  collaboration and involvement of relevant stakeholders. Government and  industry risk management guidance divides risk management activities  into four key areas—preparing for risk management, identifying and  analyzing risks, mitigating risks, and reviewing risks with executive  oversight. Table 10 describes recognized best practices in these areas.
NOAA has established policies and procedures for effective risk  management for GOES-R. For example, the program has documented a  strategy for managing risks that includes important elements, such as  relevant stakeholders and their responsibilities and the criteria for  evaluating, categorizing, and prioritizing risks. The program’s approved  risk management plan also includes requirements for risk mitigation— such as required actions, deadlines, and assigned risk owners—as well  as requirements that risks’ status and changes are periodically reviewed  by appropriate managers, including senior NOAA and NASA managers.  Table 11 compares GOES-R’s risk management policies and procedures  with recognized risk management practices for four areas.
With such policies and procedures in place, NOAA has established a  comprehensive framework to support its identification, mitigation, and  oversight of risks across the program, and laid a foundation for consistent  implementation.
subsections: section_title: The Program Has Not Fully Implemented Its Risk Management Process or Adequately Mitigated Key Risks
paragraphs: While the program has a well-defined risk management process, this  process has not been fully implemented. Table 12 identifies the extent to  which the program has implemented recognized risk management  practices through its risk process. Of particular note, the program has not  conducted and documented adequate or timely evaluations for all  potential risks—called candidate risks in the risk management plan—in its  risk list. In addition, the program did not always document its risk handling  strategies and time frames and did not always provide adequate rationale  for its decision to close risks.
The GOES-R Program Office has identified 11 risks that it considers  critical (medium- or high-level risks) and could significantly impact the  program’s development cost or schedule, documented mitigation  approaches for each risk, and tracked mitigation progress. For these and  its other risks, the program generally has mitigation plans in place that  typically include actions to be taken, deadlines, and assigned  responsibilities that could help to minimize or control the occurrence of  these critical risks. Table 12 describes the program-identified critical risks,  as of February 2012.
While the program has documented mitigation plans in place for most of  its critical risks, the program is not mitigating its most critical risk— program funding stability. Although the program has included this risk on  its top risk list and presented it to the NOAA Program Management  Council, it has not devised options for required replanning or functionality  descopes should the program experience reduced funding. Program  officials stated that the risk is external and beyond their control. However,  given that the program has made trade-off decisions regarding available  funding, functionality, and the timing of its work, it is reasonable to expect  the program to have plans in place that include possible trade-off  decisions based on different outcomes, including triggers for when  decisions need to be executed.
Further, there is at least one critical risk that is not on the program’s top  risk list that could further jeopardize the program’s launch readiness dates  and life cycle costs: GOES-S milestones may be affected by delays that  have occurred or may still occur during GOES-R development. As  discussed earlier in this report, further delays in the development of the  first satellite could result in problems for the second satellite’s scheduled  milestones since the program is planning to complete the second  satellite’s activities during periods of time set aside as reserve to  complete the first satellite’s activities. Program officials stated that a risk  has recently been added to the flight project’s risk list to reflect the  potential delays to the GOES-S development schedule. Until this risk is  added to the program’s risk list with a documented mitigation approach  and regular monitoring, NOAA could delay the analysis, planning, and  actions that would limit the impacts and occurrence of the risk, and would  thus be unprepared to face the significant consequences to the program  should this risk be realized.
While the program has well-defined policies and procedures, it has not  fully implemented its risk process. Fully implementing the recognized risk  management practices defined in GOES-R policies and procedures would  provide program officials with the assurance that all risks—including  those that are new and most critical—are adequately addressed.
section_title: Conclusions
paragraphs: NOAA has made progress toward achieving GOES-R program goals by  completing certain preliminary and critical design milestones for its flight  and ground projects. However, associated reviews, including the  programwide critical design review planned for August 2012, have been  delayed by many months. Progress was also accompanied by technical  and programmatic challenges, such as initial failures of instrument  components and replanning of ground project software releases.  Although these two specific challenges are being addressed, others such  as instrument signal problems and a 2-year cost growth of more than 30  percent for program development contracts point to troubling patterns that  will require ongoing remediation and monitoring. NOAA has allocated  budget reserves for such situations according to its established guidelines  and reports that project reserve levels are currently within those  guidelines. However, the program has used approximately 30 percent of  its reserves over the last 3 years and significant portions of development  remain for major components—including the spacecraft and Core Ground  System. In addition, the program did not change its reserves when it  restored two satellites adding approximately $3.2 billion to the program’s  baseline. Unless NOAA assesses the reserve allocations across all of the  program’s development efforts, it may not be able to ensure that its  reserves will cover ongoing challenges as well as unexpected problems  for the remaining development of all four satellites in the series.
The unreliability of the program’s integrated master schedule and some  contractor schedules adds further uncertainty to whether the program will  meet its commitments. The issues that exist among these schedules,  such as a lack of a full and consistent allocation of resources, incomplete  logic, and gaps in the critical path to project completion, are inconsistent  with our and agency best practices. NOAA has taken steps to improve  schedule reliability through more automated schedule integration, a  cross-project deliverable-tracking database, and its first schedule risk  assessment. However, unless the program addresses the full set of  scheduling weaknesses we identified, its schedules may not provide a  fully reliable basis for decision making.
NOAA has defined the policies and procedures it needs to effectively  manage and mitigate these and other program risks. Nevertheless,  officials do not have a current and comprehensive view of program risk  because these policies and procedures have not been fully implemented.  Most significantly, not all known candidate risks have been evaluated;  corrective actions have not been consistently tracked; and risks we  identified in this review are not being tracked or adequately mitigated.  Until program officials diligently execute all of the program’s defined risk  management practices and integrate these with improvements in the  management of reserves and schedules, the program is at risk of  exceeding cost and schedule targets and further slipping launch dates for  satellites in the GOES-R series.
section_title: Recommendations for Executive Action
paragraphs: To improve NOAA’s ability to execute GOES-R’s remaining planned  development with appropriate reserves, improve the reliability of its  schedules, and address identified program risks, we recommend that the  Secretary of Commerce direct the NOAA Administrator to ensure that the  following four actions are taken:
Assess and report to the NOAA Program Management Council the  reserves needed for completing remaining development for each  satellite in the series.
For all satellites in the GOES-R program, including those for which  detailed scheduling has yet to begin, address shortfalls in the  schedule management practices we identified, including but not  limited to incorporating appropriate schedule logic, eliminating  unnecessary constraints, creating a realistic allocation of resources,  ensuring an unbroken critical path from the current date to the final  satellite launch, and ensuring that all subcontractor activities are  incorporated in contractors’ integrated master schedules.
Execute the program’s risk management policies and procedures to  provide more timely and adequate evaluations and reviews of newly  identified risks, documented handling strategies for all ongoing and  newly identified risks in the risk register, time frames for when risk  mitigation and fallback plans are to be executed, adequate rationale  for decisions to close risks, and documentation and tracking of action  items from risk review board meetings or other meetings with senior  NOAA and NASA managers through completion.
Given the potential impact to the program, add the risk that GOES-S  milestones may be affected by GOES-R development to the  program’s critical risk list, and ensure that this risk and the program- identified funding stability risk are adequately monitored and  mitigated.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from the Secretary  of Commerce, who transmitted NOAA’s comments. The department  concurred with three of our recommendations and partially concurred with  one recommendation. It also provided general comments, which are  addressed below, and technical comments, which we have incorporated  into our report, as appropriate. A copy of NOAA’s comments is provided  in appendix II.
NOAA concurred with our first recommendation to assess and report on  the reserves needed for completing the remaining development for each  satellite in the series to the agency’s Program Management Council. It  stated that the GOES-R program would continue to provide status reports  on contingency reserves to the Goddard Space Flight Center and the  Program Management Council, and would work with NOAA to ensure  contingency reporting meets its needs. It also stated in its general  comments that contingency reserve for GOES-T and GOES-U  (amounting to 20 percent of their projected development costs) was  included when the $3.2 billion for these satellites was added to the  program budget baseline in February 2011. NOAA did not provide any  additional data on its fiscal year 2012 contingency budget allocations by  satellite to support this statement and, as we discussed in this report, the  program did not report a significant change in overall program reserve  levels when it revised the baseline from two satellites to four satellites.
NOAA concurred with our second recommendation to address shortfalls  that we identified in schedule management practices and stated that the  program will continue to bring down the number of errors in the schedules  and improve the fidelity of the program’s integrated master schedule.
NOAA partially concurred with our third recommendation to fully execute  the program’s risk management policies and procedures, to include timely  review and disposition of candidate risks. NOAA stated that it did not  consider the “concerns” listed in its risk database to be risks or candidate  risks, and that the risk management board actively determines whether  recorded concerns should be elevated to a risk. However, the program  has not treated concerns in accordance with its risk management plan,  which considers these to be “candidate risks” and requires their timely  review and disposition, as evidenced by the many concerns in the  database that were more than 3 months old and had not been assessed  or dispositioned. Unless NOAA follows its risk management plan by  promptly evaluating “concerns,” it cannot ensure that it is adequately  managing the full set of risks that could impact the program.
NOAA concurred with our fourth recommendation to add to the program’s  critical risk list that GOES-S milestones may be affected by GOES-R  development and to ensure that this risk and the program-identified  funding stability risk are adequately monitored and mitigated. NOAA  stated that it currently has identified and reported on the GOES-S  milestone risk and that the program is working with NOAA to monitor and  mitigate the funding stability risk.
In a general comment on our discussion of contractors’ estimated cost  growth, NOAA stated that certain growth in contract costs was due to  scope changes, including instrument options that were exercised in 2010  and 2011, and stated that such changes should be distinguished from  cost growth due to efforts to resolve problems. We consider scope  changes to be a valid component of program cost growth. Nevertheless,  we revised our report to state that approximately $60 million of the $757  million growth in contractors’ estimated costs at completion from January  2010 through November 2011 was due to scope changes associated with  new instrument flight model development. If these scope changes are  excluded, contractors’ estimated costs for flight and ground project  components still grew by approximately $700 million, or approximately 30  percent, between January 2010 and January 2012.
NOAA commented that the GOES-R launch readiness date changed from  December 2014 to October 2015 due to a protest against the spacecraft  contract award. We acknowledged the reasons for delays in GOES-R’s  launch readiness date in our draft report submitted to NOAA and in our  prior reports on this program, and have added language to this report that  relates to the bid protest and NASA’s response in delaying the award.
NOAA also commented that the revised incremental schedule was less  risky than its original waterfall schedule. While we acknowledged the  reduced risk in our draft report and have further revised our report to  reflect NOAA’s comments on our draft, we also note that the revised  development methodology introduced other risks to the program—such  as additional contractor staff and software development and verification  activities that require government oversight and continuous monitoring.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 1 day from the  report date. At that time, we will send copies to interested congressional  committees, the Secretary of Commerce, the Administrator of NASA, the  Director of the Office of Management and Budget, and other interested  parties. The report also will be available at no charge on the GAO website  at http://www.gao.gov.
If you or your staffs have any questions on the matters discussed in this  report, please contact me at (202) 512-9286 or pownerd@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) assess the National Oceanic and Atmospheric  Administration’s (NOAA) progress in developing the Geostationary  Operational Environmental Satellite-R series (GOES-R) program, (2)  evaluate whether the agency has a reliable schedule for executing the  program, and (3) determine whether the program is applying best  practices in managing and mitigating its risks.
To assess progress in developing the GOES-R satellite program, we  compared the program’s planned completion dates for key milestones  identified in its management control plan and system review plan against  actual and current estimated completion dates of milestones. We  analyzed program monthly status briefings to identify the current status  and recent development challenges of flight and ground project  components and instruments. We also analyzed contractor- and program- reported data of development costs and reserves. Finally, we conducted  several interviews with GOES-R program staff to better understand  milestone time frames, to discuss current status and recent development  challenges for work currently being performed on GOES-R, and to  understand how the program reports costs and reserve totals.
To evaluate whether NOAA has a reliable schedule for executing the  program, we evaluated contractor schedules to determine the extent to  which GOES-R is following our identified best practices in creating and  maintaining its schedules. We analyzed four contractor schedules—two  of which represent the overall schedules for the flight and ground projects  (spacecraft and Core Ground System), the program’s critical flight  instrument (Advanced Baseline Imager), and an instrument that had been  experiencing implementation issues at the time of our review  (Geostationary Lightning Mapper). We also populated workbooks as a  part of that analysis to highlight potential areas of strengths and  weakness in schedule logic, use of resources, task duration, float, and  task completion; analyzed programwide initiatives undertaken by GOES- R such as the Joint Cost and Schedule Confidence Level and the Giver- Receiver Intersegment Database; assessed GOES-R’s progress against  their own scheduling requirements, and interviewed government and  contractor officials regarding their scheduling practices.
To determine whether the GOES-R program is applying best practices in  managing and mitigating its risks, we analyzed the program’s risk  management plan to identify the program’s policies and procedures and  compared these to outputs from the program’s risk register and other  program risk management documentation, such as risk mitigation status  reports. We also reviewed documents such as monthly briefings to the  NOAA Program Management Council and program risk status reports to  determine how individual risks are managed and reviewed on a regular  basis. We assessed the extent to which the program’s policies and  procedures met government and industry recognized risk management  practices from the Software Engineering Institute’s Capability Maturity  Model® Integration for Acquisition (Version 1.3), the International  Organization for Standardization, the Defense Acquisition University, and  GAO, and whether the program fully implemented its policies and  procedures. We also examined the program’s risk status reports and risk  register to identify risks for which the program did not have an adequate  mitigation plan or were not being actively monitored by the program.  Finally, we interviewed GOES-R officials to gain further insight into the  program’s risk management process and the program’s application of its  process.
We primarily performed our work at NOAA and National Aeronautics and  Space Administration offices in the Washington, D.C., metropolitan area.  We conducted this performance audit from August 2011 to June 2012, in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Commerce
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, individuals making contributions  to this report included Colleen Phillips (assistant director), Paula Moore  (assistant director), Shaun Byrnes, Nancy Glover, Franklin Jackson,  Jason Lee, and Josh Leiling.