Geostationary Weather Satellites: Progress Made, but Weaknesses in Scheduling, Contingency Planning, and Communicating with Users Need to Be Addressed
Published: 2013-09-09T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: NOAA, with the aid of the National Aeronautics and Space Administration (NASA), is procuring the next generation of geostationary weather satellites. The GOES-R series is to replace the current series of satellites (called GOES-13, -14, and -15), which will likely begin to reach the end of their useful lives in 2015. This new series is considered critical to the United States' ability to maintain the continuity of satellite data required for weather forecasting through 2036.
GAO was asked to evaluate GOES-R. GAO's objectives were to (1) assess GOES-R progress and efforts to address key cost and schedule risks; (2) evaluate efforts to manage changes in requirements and whether any significant changes have recently occurred; and (3) evaluate the adequacy of GOES-R contingency plans. To do so, GAO analyzed program and contractor data, compared GOES-R schedules, requirements changes, and contingency plans to best practices by leading organizations, and interviewed officials at NOAA, NASA, and at other federal agencies that rely on GOES.
section_title: What GAO Found
paragraphs: The National Oceanic and Atmospheric Administration (NOAA) has completed the design of its Geostationary Operational Environmental Satellite-R (GOES-R) series and made progress in building flight and ground components. While the program reports that it is on track to stay within its $10.9 billion life cycle cost estimate, it has not reported key information on reserve funds to senior management. Also, the program has delayed interim milestones, is experiencing technical issues, and continues to demonstrate weaknesses in the development of component schedules. These factors have the potential to affect the expected October 2015 launch date of the first GOES-R satellite, and program officials now acknowledge that the launch date may be delayed by 6 months. A launch delay would increase the time that NOAA is without an on-orbit backup satellite. It would also increase the potential for a gap in GOES satellite coverage should one of the two operational satellites (GOES-14 or -15) fail prematurely - a scenario given a 36 percent likelihood of occurring by an independent review team.
While the GOES-R program has established a process for managing requirements changes, it has not effectively involved key satellite data users. Since 2007, the GOES-R program decided not to develop 31 of the original set of GOES products and modified specifications on 20 remaining products. For example, NOAA decreased the accuracy requirement for the hurricane intensity product and decreased the timeliness of the lightning detection product. However, key satellite data users were not fully informed about changes and did not have a chance to communicate their concerns about the impact of these changes on their operations. Until NOAA improves its communication with external satellite data users, obtains input from the users, and addresses user concerns when considering product changes, its changes could cause an unexpected impact on critical user operations.
NOAA has established contingency plans for the loss of its GOES satellites and ground systems that are generally in accordance with best practices; however, these plans are missing key elements. For example, NOAA did not work with the user community to address potential reductions in capability under contingency scenarios or identify alternative solutions for preventing a delay in the GOES-R launch date. Until NOAA addresses the shortfalls in its contingency plans and procedures, the plans may not work as intended in an emergency and satellite data users may not obtain the information they need to perform their missions.
section_title: What GAO Recommends
paragraphs: GAO is recommending that NOAA address weaknesses in managing reserves and scheduling, improve communications with satellite data users, and address shortfalls in contingency planning. NOAA concurred with GAO's recommendations and identified steps it is taking to implement them.

section_title: Letter
paragraphs: Geostationary environmental satellites play a critical role in our nation’s  weather forecasting. These satellites—which are managed by the  Department of Commerce’s National Oceanic and Atmospheric  Administration (NOAA)—provide information on atmospheric, oceanic,  climatic, and solar conditions that help meteorologists observe and  predict regional and local weather events. They also provide a means of  identifying the large-scale evolution of severe storms, such as forecasting  a hurricane’s path and intensity.
NOAA, through collaboration with the National Aeronautics and Space  Administration (NASA), is procuring the next generation of geostationary  weather satellites, called the Geostationary Operational Environmental  Satellite–R (GOES-R) series. The GOES-R series consists of four  satellites and is to replace the current series of geostationary  environmental satellites as they reach the end of their useful lives. This  new series is expected to provide the first major improvement in the  technology of GOES instruments since 1994 and, as such, is considered  critical to the United States’ ability to maintain the continuity of data  required for weather forecasting through the year 2036.
This report responds to your request that we review NOAA’s GOES-R  series program (GOES-R program). Specifically, our objectives were to  (1) assess GOES-R progress and efforts to address key cost and  schedule risks that we identified in our prior report, (2) evaluate efforts to  manage changes in requirements and whether any significant changes  have recently occurred, and (3) evaluate the adequacy of GOES-R  contingency plans. To assess NOAA’s progress in developing GOES-R  and addressing key risks, we compared estimated and actual program  deliverables and analyzed monthly program status briefings to identify  current status and recent development challenges. We also followed up  on our prior concerns regarding reserve funds and scheduling practices  by comparing the program’s current level of reserve funding and two  component schedules to best practices. By recalculating reserve  percentages based on supporting data and examining schedule  anomalies through use of a standard template, we determined data in  both areas to be reliable for the purposes of this audit. To assess NOAA’s  efforts to manage changes in requirements, we compared the agency’s  policies and practices to best practices identified by leading  organizations and identified major changes to the program over time. To  evaluate the adequacy of the GOES-R contingency plan, we compared  the GOES-R contingency plan to best practices in contingency planning  identified by leading organizations. We also interviewed program officials  as well as key internal and external satellite data users.
We conducted this performance audit from October 2012 to September  2013 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. See appendix I for a  complete description of our objectives, scope, and methodology.
subsections: section_title: Background
paragraphs: Since the 1970s, geostationary satellites have been used by the United  States to provide meteorological data for weather observation, research,  and forecasting. NOAA’s National Environmental Satellite, Data, and  Information Service is responsible for managing the civilian operational  geostationary satellite system, called GOES. Geostationary satellites can  maintain a constant view of the earth from a high orbit of about 22,300  miles in space.
NOAA operates GOES as a two-satellite system that is primarily focused  on the United States (see fig. 1). These satellites provide timely  environmental data about the earth’s atmosphere, surface, cloud cover,  and the space environment to meteorologists and their audiences. They  also observe the development of hazardous weather, such as hurricanes  and severe thunderstorms, and track their movement and intensity to  reduce or avoid major losses of property and life. The ability of the  satellites to provide broad, continuously updated coverage of atmospheric  conditions over land and oceans is important to NOAA’s weather  forecasting operations.
To provide continuous satellite coverage, NOAA acquires several  satellites at a time as part of a series and launches new satellites every  few years (see table 1). NOAA’s policy is to have two operational  satellites and one backup satellite in orbit at all times.
Four GOES satellites–GOES-12, GOES-13, GOES-14, and GOES-15– are currently in orbit. Both GOES-13 and GOES-15 are operational  satellites with GOES-13 covering the eastern United States and GOES- 15 in the western United States (see fig. 1). GOES-14 is currently in an  on-orbit storage mode and available as a backup for the other two  satellites should they experience any degradation in service. GOES-12 is  at the end of its service life, but it is being used to provide limited  coverage of South America. The GOES-R series is the next generation of  satellites that NOAA is planning. The first two satellites in the series  (called GOES-R and GOES-S) are planned for launch in October 2015  and February 2017, respectively.
Each of the operational geostationary satellites continuously transmits  raw environmental data to NOAA ground stations. The data are  processed at these ground stations and transmitted back to the satellite  for broadcast to primary weather services and the global research  community in the United States and abroad. Raw and processed data are  also distributed to users via ground stations through other communication  channels, such as dedicated private communication lines and the  Internet. Figure 2 depicts a generic data relay pattern from a  geostationary satellite to the ground stations and commercial terminals.
NOAA established the GOES-R program to develop and launch the next  series of geostationary satellites and to ensure the continuity of  geostationary satellite observations. Since its inception, the GOES-R  program has undergone several changes in cost and scope. As originally  envisioned, GOES-R was to encompass four satellites hosting a variety of  advanced technology instruments and providing 81 environmental  products. The first two satellites in the series were expected to launch in  September 2012 and April 2014. However, in September 2006, NOAA  decided to reduce the scope and technical complexity of the GOES-R  program because of expectations that total costs, which were originally  estimated to be $6.2 billion, could reach $11.4 billion. Specifically, NOAA  reduced the minimum number of satellites from four to two, cancelled  plans for developing an advanced instrument (which reduced the number  of planned satellite products from 81 to 68), and divided another  instrument into two separate acquisitions. The agency estimated that the  revised program would cost $7 billion and kept the planned launch dates  unchanged.
Subsequently, NOAA made several other important decisions about the  cost and scope of the GOES-R program. In May 2007, NOAA had an  independent cost estimate completed for the GOES-R program. After  reconciling the program office’s cost estimate of $7 billion with the  independent cost estimate of about $9 billion, the agency established a  new program cost estimate of $7.67 billion. This was an increase of $670  million from the previous estimate. The program also moved the launch  dates for the first two satellites to December 2014 and April 2016.  Further, in November 2007, to mitigate the risk that costs would rise,  program officials decided to remove selected program requirements from  the baseline program and treat them as contract options that could be  exercised if funds allowed. These requirements included the number of  products to be distributed, the time to deliver the remaining products  (product latency), and how often these products would be updated with  new satellite data (refresh rate). For example, program officials eliminated  the requirement to develop and distribute 34 of the 68 envisioned  products, including low cloud and fog, sulfur dioxide detection, and cloud  liquid water. Program officials included the restoration of the requirements  for the products, latency times, and refresh rates as options in the ground  system contract that could be acquired at a later time. Program officials  later reduced the number of products that could be restored as a contract  option (called option 2) from 34 to 31 because they determined that two  products were no longer feasible and two others could be combined into a  single product.
In late 2009, NOAA changed the launch dates for the first two satellites to  October 2015 and February 2017, in part due to a bid protest related to  award of the spacecraft contract. More recently, NOAA restored two  satellites to the program’s baseline, making GOES-R a four-satellite  program once again. In February 2011, as part of its fiscal year 2012  budget request, NOAA requested funding to begin development for two  additional satellites in the GOES-R series—GOES-T and GOES-U. The  program estimates that the development for all four satellites in the  GOES-R series—GOES-R, GOES-S, GOES-T, and GOES-U—is to cost  $10.9 billion through 2036, an increase of $3.2 billion over its prior life  cycle cost estimate of $7.67 billion for the two-satellite program. See table  2 for an overview of key changes to the GOES-R program.
The GOES-R program is divided into flight and ground projects that have  separate areas of responsibility and oversee different sets of contracts.  The flight project, which is managed by NASA, includes instruments,  spacecraft, launch services, satellite integration, and on-orbit satellite  initialization. Table 3 summarizes the GOES-R instruments and their  planned capabilities.
The ground project is directed by NOAA and is made up of three main  components: the core ground system, an infrastructure of antennas, and  a product access subsystem. In turn, the core ground system comprises  four functional modules supporting operations, product generation,  product distribution, and configuration control. Key components of the  ground project are described in table 4.
NOAA is responsible for GOES-R program funding and overall mission  success. The NOAA Program Management Council, which is chaired by  NOAA’s Deputy Undersecretary, is the oversight body for the GOES-R  program. However, since it relies on NASA’s acquisition experience and  technical expertise to help ensure the success of its programs, NOAA  implemented an integrated program management structure with NASA for  the GOES-R program (see fig. 3). NOAA also located the program office  at NASA’s Goddard Space Flight Center.
In recent years, we issued a series of reports aimed at addressing  weaknesses in the GOES-R program. Key areas of focus included (1)  improving communications with external data users, (2) developing  contingency plans, and (3) addressing key cost and schedule risks.
Improving communications with external users. In September  2010, we reported that while NOAA had identified GOES data users  and involved internal NOAA users in developing and prioritizing  GOES-R requirements, it had not adequately involved other federal  users who rely on GOES data by documenting their input and  communicating major changes to the program. We recommended  that the program establish processes for satellite data requirements  definition and prioritization that include documented input from  external users, as well as processes to notify these non-NOAA  agencies of GOES-R program status and changes. In February 2012,  the GOES-R program developed a communications plan that  described how external stakeholders would be notified of GOES-R  progress, status changes, and other relevant activities. However,  NOAA has not yet fully implemented the plan, as demonstrated by the  communication shortfalls discussed later in this report.
Developing contingency plans. In September 2010, we reported  that while there was a potential gap in backup coverage due to  satellite launch delays, NOAA had not established adequate continuity  plans for its geostationary satellites. We recommended that the  program’s plan include implementation procedures, resources, staff  roles, and timetables needed to transition to a single satellite, a  foreign satellite, or other solution. In December 2012, NOAA finalized  a contingency plan that generally included these elements. However,  more work remains to ensure that the plan is viable.
More recently, in February 2013, we added the potential gaps in  weather satellite data to our biennial High-Risk list. In that report, we  noted that NOAA had established a contingency plan for a potential  gap in the GOES program, but it needed to demonstrate its progress  in coordinating with the user community to determine their most  critical requirements, conducting training and simulations for  contingency operations scenarios, evaluating the status of viable  foreign satellites, and working with the user community to account for  differences in product coverage under contingency operations  scenarios. We also stated that NOAA should update its contingency  plan to provide more details on its contingency scenarios, associated  time frames, and any preventative actions it is taking to minimize the  possibility of a gap.
Addressing key cost and schedule risks. In June 2012, we  reported that the GOES-R program might not be able to ensure that it  had adequate resources to cover unexpected problems in remaining  development, and that unresolved schedule deficiencies existed in its  integrated master schedule and contractor schedules. We also  reported that the program estimated a 48 percent chance that the  planned GOES-R launch date of October 2015 would be reached.  We recommended that the program assess and report on the  reserves needed for completing remaining development for each  satellite in the series, and address shortfalls in the schedule  management practices we identified such as eliminating unnecessary  constraints and creating a realistic allocation of resources, in order to  minimize the likelihood of a potential gap. The agency agreed with  these recommendations and took steps to address them by identifying  needed reserve levels and refining program schedules.
section_title: NOAA Has Made Progress in Developing GOES-R, but Continues to Face Challenges that Could Increase the Risk of a Satellite Data Gap
paragraphs: NOAA has completed its design of the GOES-R program, and has made  progress in building components of the flight and ground segments.  Program officials also report that the program is operating within its  estimated budget of $10.9 billion. However, key information on reserves  has not been reported to management. Further, both the flight and ground  segments have experienced delays in achieving major milestones due to  technical challenges, and weaknesses in the development of master  schedules could cause further delays. Program officials stated that they  have made improvements on how they manage cost reserves and  schedules, but acknowledged that there will always be opportunities for  improvement because the reserves and schedules are so dynamic on a  big program like GOES-R. These challenges have the potential to impact  the expected launch date of the first GOES-R satellite, which would delay  the availability of an on-orbit backup and increase the potential for a gap  in GOES satellite coverage should either of the two operational satellites  fail prematurely.
subsections: section_title: Program Has Completed Design and Begun Building Components of the First Satellite
paragraphs: NASA and NOAA are following NASA’s standard space system life cycle  on the GOES-R program. This life cycle includes distinct phases,  including concept and technology development; preliminary design and  technology completion; final design and fabrication; system assembly,  integration and testing, launch and checkout; and operations and  sustainment. There are key program reviews throughout each of the  phases, including preliminary design review, critical design review, and  system integration review. NOAA and NASA jointly conduct key reviews  on the flight and ground segments individually as well as for the program  as a whole, and then make decisions on whether to proceed to the next  phase. Figure 4 provides an overview of the life cycle phases, key  program reviews, and associated decision milestones. In addition, the key  reviews are described in table 5.
The GOES-R program has completed final design and begun building  components of the flight and ground systems. Specifically, the program  completed critical design reviews for the flight and ground projects and for  the overall program between April and November 2012. In its evaluation  of the program as part of the critical design review, an independent  review board complimented the program on several recent achievements,  stating that the program was beyond the level of maturity expected at that  phase, and that the program’s planning was a major factor in the launch  date of the first satellite remaining October 2015.
As the spacecraft and instruments are developed, NASA conducts  several interim reviews and tests before proceeding to the next major  program-level review, the system integration review. These include a pre- environmental review, which represents the conclusion of an initial round  of testing before exposing the instrument to testing under adverse  environmental conditions; environmental testing of key functions under  adverse conditions; and a pre-shipment review, which is conducted on  each instrument to ensure it is ready to be shipped for integration and  testing on the spacecraft.
The GOES-R flight components are in various stages leading up to the  system integration review. Of the six GOES-R instruments, one has  completed environmental testing and its pre-shipment review; four  instruments are in the midst of these reviews and tests; and one  instrument has not yet passed its pre-environmental review. In addition,  the program began building the spacecraft in February 2013. Table 6  provides more information on progress made on the key flight project  components.
Similar to the flight project, major ground system milestones are focused  on building and testing components and the program has made progress  in this area. Specifically, on the core ground system, a prototype for the  operations module was delivered in late 2012 and used for initial testing  and training. In July 2013, the ground project delivered the iteration of  the operations module that will be used to support the first satellite. In  addition, the program has installed antenna dishes at NOAA’s primary  satellite communications site, and completed two key reviews of antennas  at the GOES remote backup site. The Product Distribution and Access  System recently completed a review that will allow testing to begin on its  first release. An integration review for ground components is also  expected to take place in January 2014. More detail on the progress of  the ground project can be seen in table 7.
The program’s next major milestone is a programwide system integration  review, which is scheduled for March 2014. Based on the results of that  review, NOAA and NASA will decide whether to move the program to the  next phase: the system assembly, integration and test, and launch and  checkout phase.
section_title: Contingency Reserves Are Generally in Line with Goals for Overall Program Development; Reporting on Reserve Values Remains Limited
paragraphs: The GOES-R program is estimated to cost $10.9 billion. As of February  2013, the program estimated that this amount was divided into four  categories, with $6.0 billion for the flight project, $1.7 billion for the ground  project, $2.0 billion for other program costs (including, among other  things, program/project support and formulation) and $1.2 billion for  operations and support. Program officials reported that the program is  currently operating without cost overruns on any of its main components,  but noted that the program life cycle costs may increase by $150 to $300  million if full funding in the current fiscal year is not received.
A portion of the amounts planned for the flight project and ground project  are allocated to contingency reserves (also called management reserve).  The program also keeps a programwide contingency allocation separate  from those of the flight and ground projects. A contingency reserve  provides program managers ready access to funding in order to resolve  problems as they occur and may be necessary to cover increased costs  resulting from unexpected design complexity, incomplete requirements, or  other uncertainties. NASA’s Goddard Space Flight Center requires its  flight projects, including GOES-R, to maintain contingency reserves  during system development, at a level of 25 percent of development  costs. The GOES-R program requires its flight and ground projects to  maintain 20 percent of planned remaining development costs as reserve  funding. The program office also maintains contingency reserves equal  to 10 percent of planned remaining development costs to cover program  support costs and to supplement the flight and ground projects’ reserves  if necessary. According to a NOAA program official, the GOES-R program  is able to meet NASA’s requirement through the combination of the 20  percent flight and ground project requirements and the supplemental 10  percent program-level reserve. An official also stated that the method of  keeping separate reserves at the program and project levels was chosen  as it was successful on past projects.
The GOES-R flight project, ground project, and program office are at or  above the amount of reserves they are required to carry. Specifically, as  of March 2013, the overall contingency reserve percentages for the flight  and ground projects were at 20 and 28 percent, respectively, which are at  or above the required level of 20 percent. The program reserves were at  11 percent, slightly above the required level of 10 percent. Reserve  values and percentages are provided in table 8.
We previously reported that, in order to oversee GOES-R contingency  funding, senior managers should have insight into the amount of reserves  set aside for each satellite in the program and detailed information on  how reserves are being used on both the flight and ground components.  While the GOES-R program continues to regularly report on contingency  funds, it does not report key information on the status of its reserve  funding to senior level management.
At monthly program management council meetings, the program reports  summary information, such as the total value of contingency reserves and  reserve percentage held for each fiscal year. Reserve totals are given for  the flight and ground projects as well as for the program overall. However,  the program does not report on the reserves needed for completing  remaining development for each satellite in the series or provide detailed  information on how reserves are being used. Thus, for example, if the  later satellites in the series have a high level of reserves, they could mask  low reserve values for earlier satellites in the series. Further, in its  monthly presentations to senior managers and NOAA executives, the  program does not include information on the cause of any changes in  reserve values from the prior month or the assumptions it makes when  calculating reserves. For example, the flight reserve value recently went  up by 2 percentage points because the program decided to include  reserve funding for the GOES-T satellite in 2018, and the ground reserve  values went down by 10 percentage points because the program shifted  reserve funding from the ground to the flight projects. Neither of these  changes was identified or explained in the monthly presentations. The  lack of insight on how the reserves are calculated and modified could lead  executives to misinterpret the size of the remaining reserves. Program  officials noted that they took steps after our previous report to clarify what  they report about reserves, but noted that the amount of information  needed to fully explain reserve calculations and changes could be too  much information for an executive-level briefing. Without regularly  providing sufficiently detailed budget information, it may be more difficult  for program management to have the information they need to make the  best decisions possible regarding the program’s future funding.
section_title: Recent and Potential Milestone Delays and Continued Weaknesses in Scheduling Practices Increase the Potential for a Delayed Launch
paragraphs: The GOES-R program established programwide milestones, including the  mission operations review and flight operations review, to determine the  program’s ability to proceed to system integration and to complete  mission operations, respectively. It also established five end-to-end  system tests to validate compatibility between the space and ground  segments before the launch of the first satellite.
However, over the past year, the program delayed many of these key  milestones and tests. Delays in the mission operations review means that  the large-scale integration of flight and ground components will not occur  until 21 months prior to launch. Similarly, delaying end-to-end tests until  17 months prior to launch will allow the program less time to respond to  any problems that occur. Table 9 highlights key milestones and the extent  of recent delays.
The GOES-R program is also experiencing technical issues on the flight  and ground projects that could cause further schedule delays.
The original supplier for a key component on the spacecraft moved to  a different facility, introducing risk due to the loss of experienced  personnel and the impact on schedule. This led the program to find an  alternative supplier. While a design review was performed to confirm  resolution of the issue in April 2013, this change may lead to a delay  of up to 6 months in integrating the component on the spacecraft.  Program officials noted that this delay is not expected to impact the  program’s critical path or major milestones.
The Geostationary Lightning Mapper’s electronics unit experienced  problems during testing, which led the program office to delay the  tests. The program is considering several options to address this  issue, including using the electronics unit being developed for a later  GOES satellite to allow key components to proceed with testing. If the  issue cannot be resolved, it would affect the instrument’s  performance. As a result, the program is also considering excluding  the Geostationary Lightning Mapper from the first GOES satellite. The  program plans to make its decision on whether or not to include the  instrument in late 2013. The removal of this instrument would cause a  significant reduction in the satellite’s functionality. Key GOES users  have stated that they would prefer that NOAA delay launching the  GOES-R satellite rather than launch it without the Geostationary  Lightning Mapper.
The program delayed the start of work on the ground system at the  NOAA satellite operations facility by three months, from a planned  date of October 2012 to January 2013, following a bid protest of the  award of a contract to upgrade the facility. This delay compressed an  already tight schedule for testing the ground system.
Testing for a number of ground system requirements has been  postponed until future releases and builds, potentially causing  modification to the schedule for these future products.
Power amplifiers for the antenna systems experienced higher than  expected failure rates, which could lead to schedule delays and  decreases in operational availability.
Given that fewer than 3 years remain before GOES-R’s expected launch  in October 2015, continued delays in key milestones and reviews  decrease the likelihood that the launch date will be met. Program officials  recently acknowledged that the GOES-R launch date may be delayed by  about 6 months, and attributed the cause of the delay to a shortfall of $54  million in anticipated funding in fiscal year 2013.
Delays in the program’s remaining schedule are also at risk of further  growth due to weaknesses in the program’s scheduling methods.  Program schedules not only provide a road map for systematic program  execution, but also provide the means by which to gauge progress,  identify and address potential problems, and promote accountability.  Achieving success in managing large-scale programs depends in part on  having an integrated and reliable schedule that defines, among other  things, when work activities and milestone events will occur, how long  they will take, and how they are related to one another. Without such a  reliable schedule, program milestones may slip.
In June 2012, we reported on weaknesses in program schedules that  comprised portions of the program’s Integrated Master Schedule,  including subordinate schedules for the spacecraft and core ground  system. At that time, our work identified nine best practices associated  with developing and maintaining a reliable schedule. These are (1)  capturing all activities, (2) sequencing all activities, (3) assigning  resources to all activities, (4) establishing the duration of all activities, (5)  integrating schedule activities horizontally and vertically, (6) establishing  the critical path for all activities, (7) identifying reasonable float time  between activities, (8) conducting a schedule risk analysis, and (9)  updating the schedule using logic and durations. See table 10 for a  description of each of these best practices.
In a previous report, we observed that important schedule components in  GOES-R related schedules had not been included or completed, and  recommended that these shortfalls be addressed. NOAA has since  improved selected practices on its spacecraft and core ground schedules,  but other practices stayed the same or worsened. Specifically, for the  spacecraft, 2 practices were improved, 5 stayed the same, and 2 became  weaker. For the core ground system, 4 practices were improved, 3 stayed  the same, and 2 became weaker. Table 11 compares our assessments of  the spacecraft and core ground system schedules in July 2011 and  November 2012.
NOAA has improved elements of the schedules for both components.  Specifically, the spacecraft schedule has eliminated level of effort  activities and has assigned resources for a greater percentage of  activities. The core ground schedule now has an automated process by  which all subcontractor records are combined to create an integrated  schedule. It has a series of connected activities that lead to what  contractor officials consider its main milestone delivery, and has  implemented a detailed schedule risk analysis for a key upcoming  release.
However, scheduling issues remain on the schedules for both  components. For example, both schedules have issues with sequencing  remaining activities and integration between activities. Regarding the  spacecraft schedule, there is a small subset of activities with incomplete  links between activities, and more than 20 percent of remaining detail  activities have lags, or a set number of days between an activity and its  successor. In the core ground schedule, a number of activities are  missing either predecessor or successor activities, and there are several  activities representing the end of the project on or about the same date.  Without the right linkages, activities that slip early in the schedule do not  transmit delays to activities that should depend on them. When this  happens, the schedule will not provide a sufficient basis for understanding  the program as a whole, and users of the schedule will lack confidence in  the dates and the critical path.
Both schedules also have a very high average of total float time for  detailed activities. Specifically, total float time is greater than two months  for nearly two-thirds of remaining detailed activities in the spacecraft  schedule, and at least a year for more than 10 percent of remaining detail  activities in the core ground schedule. In the case of spacecraft, officials  stated that high levels of float time were often due to activities that had  been completed at one time for several satellites, only one of which was  immediately needed. Officials also provided detailed information on the  activities with the highest amount of float. In the case of the core ground  schedule, officials stated that many activities occurring after the main  milestone date, which occurs nearly five years prior to the end of the  schedule, do not have a true successor, and therefore are calculated only  to the end of the contract. Officials also stated that values and trends in  float time are monitored regularly for both schedules. Such high values of  total float time can falsely depict true project status, making it difficult to  determine which activities drive key milestone dates. Without reasonable  values of total float time, it cannot be used to identify activities that could  be permitted to slip and thus release and reallocate resources to activities  that require more resources to be completed on time.
In addition, the project’s critical path does not match up with activities that  make up the driving path on the core ground schedule. Contractors  monitor a driving path monthly to both major and minor milestone  deliveries. However, until the schedule can produce a true critical path, it  will be more difficult for the program office to provide reliable time line  estimates or identify when problems or changes may occur and their  effect on downstream work. Also, without a valid critical path to the end of  the schedule, management cannot focus on activities that will have a  detrimental effect on the key project milestones and deliveries if they slip.
Further, neither schedule file has fully integrated resources with schedule  activities. As of November 2012, contractor officials stated that the ground  system schedule was not feasible given available resources and that they  were in the process of revising their immediate schedules to make them  feasible. The spacecraft schedule contains major resource categories that  correspond to contractor sites and work phases. However, thresholds for  overruns of resource allocations are functionally disabled within the  schedules through the setting of an arbitrarily high value for maximum  resources per category. In response, contractor officials stated that  account managers are responsible for monitoring resource levels and that  weekly meetings are held to ensure that resource issues are discussed.  Information on resource needs and availability in each work period assists  the program office in forecasting the likelihood that activities will be  completed as scheduled. If the current schedule does not allow insight  into current or project allocation of resources, then the risk of delays in  the program’s schedule is significantly increased.
Deficiencies in scheduling practices such as the ones outlined here could  increase the likelihood of launch date delays, because decision making  would be based on data that does not accurately depict current status,  thus impeding management’s ability to conduct meaningful oversight on  the program’s schedules. Program officials noted that they have made  improvements in scheduling practices, but explained that because the  schedules are so dynamic there are always areas for improvement. Lack  of the proper understanding of current program status due to schedules  that are not fully reliable undercuts the ability of the program office to  manage a high-risk program like GOES-R.
section_title: Delays in the GOES-R Launch Date Could Increase the Risk of a Satellite Data Gap
paragraphs: Potential delays in the launch date of the first GOES-R satellite would  increase the risk of a gap in GOES satellite coverage. NOAA’s policy is to  have two operational satellites and one backup satellite in orbit at all  times. This policy proved useful in December 2008 and again in  September 2012, when the agency experienced problems with one of its  operational satellites, but was able to move its backup satellite into place  until the problems had been resolved.
NOAA is facing a period of at least a year when it will not have a backup  satellite in orbit. Specifically, in April 2015, NOAA expects to retire one of  its operational satellites (GOES-13) and move its backup satellite (GOES- 14) into operation. Thus, the agency will have only two operational  satellites in orbit—and no backup satellite—until GOES-R is launched  and completes an estimated 6-month post-launch test period. If GOES-R  is launched in October 2015, the soonest it could be available for  operational use would be April 2016. Any delay to the GOES-R launch  would extend the time without a backup to more than one year. Figure 5  shows anticipated operational and test periods for the two most recent  series of GOES satellites.
In addition to the year or more during which no back-up satellite would be  available, there is a chance that NOAA would have to operate with a  single operational satellite. In December 2012, an independent review  board estimated that there is a 36 percent chance that the GOES  constellation would have only one operational satellite at the expected  date of GOES-R’s launch. Thus, if NOAA were to experience a problem  with either of its operational satellites before GOES-R is in orbit and  operational, it would need to rely on older satellites that are beyond their  expected operational lives and may not be fully functional. Without a full  complement of operational GOES satellites, the nation’s ability to  maintain the continuity of data required for effective weather forecasting  could be compromised. This, in turn, could put the public, property, and  the economy at risk.
section_title: NOAA Has a Process for Managing Changes in GOES-R Requirements, but Changes Could Affect Some Users
paragraphs: System requirements describe the functionality needed to meet user  needs and perform as intended in an operational environment. According  to leading industry, academic, and government entities, a disciplined  process for developing and managing requirements can help reduce the  risks of developing or acquiring a system. One key aspect of effective  requirements management involves managing changes to requirements  through a standardized process. Table 12 outlines best practices of a  sound change management process and key questions for evaluating the  process.
The GOES-R program has a change management process that satisfied  three practices, partially satisfied two practices, and did not satisfy one  practice. Specifically, GOES-R has established a change management  process that tracks and documents changes in requirements, documents  the rationale for the changes as well as the potential impact of the change  on cost and schedule, and ensures that changes are reviewed and  approved by a change control board. In addition, the program has  evaluated the impact of key changes on selected users and  communicated with those users. However, as we first reported in 2010,  the program is still weak in evaluating the impact of changes on external  users who rely on GOES data products and in effectively communicating  changes to those satellite data users. Specifically, table 13 outlines how  the GOES-R program performed on each of the best practices for  managing changes in requirements, and is followed by a more detailed  discussion of key shortfalls.
While the program generally communicates requirements changes to key  users within NOAA’s National Weather Service community, it does not  communicate as well with satellite data users external to NOAA. Many  such users are dependent on GOES satellite data for their respective  missions. Officials responsible for working with satellite weather  observations at three agencies were unaware of selected changes in  GOES-R requirements. For example, the Federal Aviation Administration  uses the satellites’ data and images to manage air traffic across the  country, and the Navy uses the data for oceanic weather forecasting, as  well as tactical ocean analysis of regions of interest. They stated that  NOAA had not reached out to them to alert them to these changes or ask  if the changes would impact them. Similarly, Forest Service officials were  concerned that potential changes in spectrum allocations could affect  their ability to obtain data from their own ground-based weather  observation systems because they currently rely on GOES-R  communication channels to obtain this data.
GOES-R program officials noted that they provide regular briefings to the  Office of the Federal Coordinator for Meteorology, an interagency council  with membership from fifteen federal departments and agencies involved  in meteorological activities (including the Departments of Agriculture,  Defense, and Transportation) and that the Air Force represents the  Department of Defense community on the GOES-R Series Independent  Advisory Committee. However, they acknowledged that they cannot  ensure that the information they provide is disseminated within the  agencies. Further, GOES officials explained that one reason for the  distinction between the internal and external users is that the internal  users belong to formal working groups and receive regular updates from  the GOES-R program, while the other users generally have more informal  or indirect connections with the program. Instead of direct  communications such as e-mails, the other users may receive information  about GOES-R requirements changes from publically available  information or through other meteorological partnerships with NOAA.  Without consistent and direct communication, users may be unaware of  changes to program requirements that are critical to their respective  missions. Because GOES-R users across the country have missions that  preserve and protect property and life, it is critical that these organizations  are made aware of any changes as soon as they are made, so that they  can assess and mitigate any significant impacts.
Since 2007, NOAA has changed multiple system requirements on the  GOES-R program. These changes involved strengthening or relaxing  specifications on selected products, finalizing a decision not to develop 31  products, and modifying programmatic requirements not tied to any  individual product. For example, NOAA strengthened specifications for  the geographic coverage, image resolution, and refresh rate on a product  depicting total precipitable water, and strengthened accuracy  specifications for a product depicting cloud layers and heights. NOAA  also relaxed specifications to provide less measurement accuracy on a  product depicting hurricane intensity, less geographic coverage on a  product depicting sea surface temperatures, less resolution on a product  that tracks the motion of clouds through the atmosphere, and less timely  updates on a product depicting lightning detection. The GOES-R program  also documented NOAA’s earlier decision not to develop or provide 31  products that it labeled as optional, noting that the products will only be  developed if funding becomes available. In addition, programmatic  changes include the elimination of 97 percent mission availability as a  measure of minimum performance success and the decision not to  transmit raw satellite data to users. Table 14 provides an overview of key  changes in product and program requirements since 2007.
While NOAA officials stated that they believe that only one of the changes  that were made since 2007 was significant, internal and external users  noted that they found many of the changes to be significant. In addition,  selected satellite data users noted concern at the loss of 17 of the  optional products that are no longer being developed. The changes that  users found significant, along with user reasoning for why these changes  are significant, are listed in table 15. GOES-R program officials  acknowledged that the National Weather Service and other users will  have impacts from the loss or degradation of products, but that it is not  always accurate to assume that GOES-R could have met the original  requirements. In 2011, an algorithm development executive board  reported that several original requirements could not have been met  because, among other reasons, they relied on a hyperspectral instrument  that was removed from the program, the requirements were poorly stated  and it only became evident later that GOES could not support them, and  there were scientific limitations on the development of the products that  only became evident after development had started. Program officials  stated that they have identified alternative methods for obtaining certain  products (some outside the scope of GOES-R) and that they are  proactively trying to develop alternative products in coordination with  users and other development organizations.
In addition to the changes that have already been implemented on the  GOES-R program, there are other potential changes that could occur. For  example, by the end of 2013, the program plans to decide whether or not  to include the Geostationary Lightning Mapper on the GOES-R satellite.  Also, there could be changes in the spectrum allocated to weather  satellite data. Officials from the National Weather Service and Forest  Service raised concerns that these potential changes could also affect  their operations. Because these changes have the potential to impact  satellite data user operations, it is critical that the GOES-R program  communicates program changes to the extended user community. By  doing so, satellite data users can establish plans to mitigate any shortfalls  in data and minimize the impact of the changes on their operations.
section_title: NOAA Developed GOES-R Contingency Plans, but Weaknesses Increase the Impact of a Potential Coverage Gap
paragraphs: GOES satellite data are considered a mission-essential function because  of their criticality to weather observations and forecasts. These  forecasts—such as those for severe storms, hurricanes, and tornadoes— can have a substantial impact on our nation’s people, infrastructure, and  economy. Consequently, NOAA policy requires that there must be two in- orbit GOES satellites and one on-orbit spare in operation at all times. If  one of the operational satellites were to fail, the on-orbit spare could be  moved into position to take the place of the failed satellite. However, if  there are delays in the launch of the GOES-R satellite or if either of the  two satellites currently in operation were to fail, NOAA would not have an  on-orbit spare to fill the gap.
Government and industry best practices call for the development of  contingency plans to maintain an organization’s essential functions in the  case of an adverse event. These practices include key elements such  as defining failure scenarios, identifying and selecting strategies to  address failure scenarios, developing procedures to implement the  selected strategies, identifying any actions needed to implement the  strategies, testing the plans, and involving affected stakeholders. These  elements can be grouped into categories, including (1) identifying failure  scenarios and impacts, (2) developing contingency plans, and (3)  validating and implementing contingency plans (see table 16).
NOAA has established contingency plans for both its GOES satellites and  its associated ground systems. In September 2010, we recommended  that NOAA develop and document continuity plans for the operation of  geostationary satellites that include the implementation procedures,  resources, staff roles, and timetables needed to transition to a single  satellite, a foreign satellite, or other solution. In September 2011, the  GOES-R program provided a draft plan documenting a strategy for  conducting operations if there were only a single operational satellite. In  December 2012, the program provided us with a final version of this plan.  It included scenarios for three, two, and one operational satellites. In  addition to this satellite contingency plan, NOAA has another  contingency-related plan with activation procedures for its satellites.
Furthermore, the NOAA office responsible for ground-based satellite  operations and products has created plans for contingency operations at  the GOES ground system facility, the Satellite Operations Control Center.  Specifically, NOAA’s plans describe the transfer of critical functions to a  backup facility during an emergency. The continuity plan contains, among  other things, descriptions of the alternate locations for resources, and the  performance of key functions and implementation procedures.
When compared to best practices, NOAA’s satellite and ground system  contingency plans had many strengths and a few weaknesses.  Specifically, the satellite contingency plan fully implemented seven  elements, partially implemented nine elements, and did not implement  one element. The ground system contingency plan fully implemented ten  elements, partially implemented six elements, and one element was not  applicable. Table 17 shows the extent to which the satellite and ground  system contingency plans fully implemented, partially implemented, or did  not implement key contingency planning elements.
NOAA has implemented most of the best practices on both the GOES  satellite and ground contingency plans. Specifically, NOAA identified  failure scenarios, recovery priorities, and minimum levels of acceptable  performance. NOAA also established contingency plans that identify  solutions and high-level activities and triggers to implement the solutions.  Further, the agency has tested its contingency plans, trained staff on how  to implement the contingency plans, and updated the plans when  warranted. The agency also successfully implemented its contingency  plans when it experienced problems with one of its operational satellites.  Specifically, when GOES-13 experienced problems in September and  October 2012, NOAA activated its contingency plans to move its back-up  satellite into position to provide observations until GOES-13 was once  again operational. While the agency has not needed to address the loss  of a back-up satellite in recent years, contingency plans cover this  situation by determining if older GOES satellites could provide coverage,  moving the single satellite into a central position over the country, and  seeking data from foreign satellites.
However, both satellite and ground contingency plans contain areas that  fall short of best practices. For example, NOAA has not demonstrated  that the contingency strategies for both its satellite and ground system are  based on an assessment of costs, benefits, and impact on users. Further,  the satellite plan does not specify procedures for working with the user  community to account for potential reductions in capability under  contingency operations. For example, officials from the Federal Aviation  Administration noted that NOAA’s contingency plans do not define the  compatibility, security, and standard protocol language they should use if  a foreign satellite were to be utilized. Also, while selected users reported  that, in the past, they have been well informed by NOAA when changes in  service occur, including the problems with GOES-13, others were either  not informed or received information on outages through a third party.  Moreover, selected users stated that certain contingency operations could  have a significant impact on their operations. For example, Federal  Aviation Administration officials stated that flight approaches in Alaska  that were enabled using the Global Positioning System were affected by  the GOES-13 outage in late 2012. As another example, Forest Service  officials explained that if GOES were to experience an outage and not  have a backup satellite available, it was their understanding that NOAA  would either move a single satellite into a central position over the country  or obtain observations from a foreign satellite. Under both of these  scenarios, they could lose views of wildland fires and their ability to obtain  data from ground-based observation networks. Nearly all users stated  that the effects of a switch to a single satellite or foreign satellite  configuration would be significant.
In addition, while NOAA’s failure scenarios for its satellite system are  based on the number of available satellites—and the loss of a backup  satellite caused by a delayed GOES-R launch would fit into these  scenarios—the agency did not identify alternative solutions or time lines  for preventing a GOES-R launch delay. According to NOAA officials, a  gap caused by a delayed launch would trigger the same contingency  actions as a failure on launch or the loss of a currently on-orbit satellite.  However, this does not take into account potential actions that NOAA  could undertake to prevent a delayed launch, such as removing selected  functionality or compressing test schedules.
NOAA officials stated that their focus on primary users and on the number  of available satellites is appropriate for their contingency plans. Given the  potential for a delay in the launch of the GOES-R satellite and the  expectation that there will be at least a year with no backup satellite in  orbit, it is important that NOAA consider ways to prevent a delay in the  GOES-R launch, and ensure its contingency plans are fully documented,  tested, and communicated to affected stakeholders. Further, it is critical  that NOAA and users are aware of how contingency scenarios will affect  user operations. Until comprehensive plans are developed, it is less  certain that NOAA can provide a consistent level of service and  capabilities in the event of an early failure or late launch. This in turn  could have a devastating effect on the ability of meteorologists to observe  and report on severe weather conditions.
section_title: Conclusions
paragraphs: The GOES-R program is well on its way toward developing the first  satellite in the series, but it continues to face risks that could delay the  first satellite’s launch. Among these risks are issues we have previously  raised on how the program manages reserve funds and implements  sound scheduling practices. Specifically, the agency does not provide  important details on its contingency reserve funds to senior executives,  including the reserves allocated for each of the four satellites or key  assumptions made in calculating reserves. Without this information,  program officials could misinterpret the size of the remaining reserves  and make poor decisions regarding the program’s future funding. The  agency has improved selected scheduling practices, but others remain  weak—in part, according to agency officials, due to the dynamic nature of  scheduling a program as complex as the GOES-R satellite program. As  the agency closes in on its expected launch date, technical issues in  developing the space and ground segments and scheduling problems  could make it more difficult to launch on schedule, and program officials  now acknowledge that the launch date may be delayed by 6 months. Any  delay in the anticipated launch date would expand a potential one-year  gap in the availability of an on-orbit backup GOES satellite, and raise the  risk of a gap in geostationary satellite data should one of the two  operational satellites experience a problem.
While the agency has made multiple changes to GOES-R requirements in  recent years, it has not effectively involved satellite data users in those  changes. Specifically, internal NOAA and external satellite data users  were not fully informed about changes in GOES-R requirements and did  not have a chance to communicate their concerns about the impact these  changes could have on their ability to perform their missions. Many of  these users expressed concerns about the effect these changes could  have on their ability to fulfill their missions, including facilitating air traffic,  conducting military operations, and fighting wildland fires. Until NOAA  improves its outreach and communication with external satellite data  users, its changes in requirements could cause unexpected impacts on  critical user operations.
Given the possibility of a gap in geostationary satellite coverage, NOAA  has established contingency plans for both its GOES satellites and  ground systems; these plans include the likely scenario in which there will  not be an on-orbit backup. While these plans include many elements  called for in industry best practices, the satellite contingency plan did not  assess the potential impacts of a failure on users, or specify actions for  working with the user community to address these potential reductions in  capability under contingency operations. They also did not identify  alternative solutions or time lines for preventing a delay in the GOES-R  launch date. The absence of a fully-tested and complete set of GOES-R- related contingency plans and procedures could have a major impact on  levels of service provided in the event of a satellite or ground system  failure.
section_title: Recommendations for Executive Action
paragraphs: To address risks in the GOES-R program development and to help  ensure that the satellite is launched on time, we are making the following  four recommendations to the Secretary of Commerce. Specifically, we  recommend that the Secretary of Commerce direct the NOAA  Administrator to:
Direct program officials to include information on the amount of  reserve funding for each of the four satellites in the program as well  as information on the calculation and use of reserves in regular  briefings to NOAA senior executives, so that executives are fully  informed about changes in reserve levels.
Given the likely gap in availability of an on-orbit GOES backup  satellite in 2015 and 2016, address the weaknesses identified in this  report on the core ground system and the spacecraft schedules.  These weaknesses include, but are not limited to, sequencing all  activities, ensuring there are adequate resources for the activities, and  conducting a schedule risk analysis.
Improve communications with internal and external satellite data  users on changes in GOES-R requirements by (a) assessing the  impact of changes on user’s critical operations; (b) seeking  information from users on any concerns they might have about past or  potential changes; and (c) disseminating information on past and  potential changes in requirements to satellite data users.
Revise the satellite and ground system contingency plans to address  weaknesses identified in this report, including providing more  information on the potential impact of a satellite failure, identifying  alternative solutions for preventing a delay in GOES-R launch as well  as time lines for implementing those solutions, and coordinating with  key external stakeholders on contingency strategies.
section_title: Agency Comments and Our Evaluation
paragraphs: We sought comments on a draft of our report from the Department of  Commerce and NASA. We received written comments on a draft of this  report from Commerce transmitting NOAA’s comments. NOAA concurred  with all four of our recommendations and identified steps that it is taking  to implement them. It also provided technical comments, which we have  incorporated into our report, as appropriate. NOAA’s comments are  reprinted in appendix II.
While NOAA concurred with our recommendation to include information  on reserve funding for each of the four satellites in the program and  information on the calculation and use of reserves in regular briefings to  senior executives, and suggested that its current processes fulfill this  recommendation, we do not believe they do. Specifically, NOAA stated  that the GOES-R program currently reports on reserve funding at two  major monthly management meetings, which alerts management if  reserves fall below designated thresholds for the remaining work on all  four satellites. The agency also stated that its reporting of the percent of  “unliened” contingency funding—the amount of contingency funding not  allocated to a potential risk or issue—for the remaining work addresses  our concern regarding whether there are sufficient reserves to complete  the GOES-R series.
However, the GOES-R program does not currently identify the reserve  funding needed for each individual satellite or provide details on how  reserves are being calculated and used at the monthly management  meetings. By not providing reserve information on the individual satellites,  the program is not alerting management about potential near-term  funding shortfalls. For example, maintaining a high level of reserves on  the later satellites could mask a low level of reserves in the near-term for  GOES-R and S. Such a scenario could affect the satellites’ development  schedules and launch dates. Further, by not obtaining details on the  assumptions made when calculating reserves and the causes of changes  in reserve values, management is unable to determine if changes in  reserve levels are due to the addition, subtraction, or use of funds, or to  changes in the assumptions used in the calculations. Given the  importance of reserve funds in ensuring the satellite development  remains on track, management should be aware of reserve funding levels  for each individual satellite and of the underlying reasons for changes in  reserve levels. Therefore, we continue to believe that additional action is  needed by NOAA to respond to our recommendation.
After we received agency comments and while our report was in final  processing, NOAA notified us that the launch dates of the first and  second GOES-R series satellites would be delayed. Given the late receipt  of this information, our report reflects the previous launch date.
NASA did not provide comments on the report’s findings or  recommendations, but noted that it would provide any input it might have  to NOAA for inclusion in that agency’s comments.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to interested congressional  committees, the Secretary of Commerce, the Administrator of NASA, the  Director of the Office of Management and Budget, and other interested  parties. The report also will be available at no charge on the GAO website  at http://www.gao.gov.
If you or your staff have any questions on the matters discussed in this  report, please contact me at (202) 512-9286 or at pownerd@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) assess the National Oceanic and Atmospheric  Administration’s (NOAA) progress in developing the Geostationary  Operational Environmental Satellite-R series (GOES-R) program and in  addressing key cost and schedule risks that we identified in a prior report,  (2) evaluate the program’s efforts to manage changes in requirements  and whether any significant changes have recently occurred, and (3)  evaluate the adequacy of GOES-R contingency plans.
To assess NOAA’s progress in developing the GOES-R satellite program,  we compared the program’s planned completion dates for key milestones  identified in its management control plan and system review plan against  actual and currently estimated completion dates. We analyzed monthly  program status briefings to identify the current status and recent  development challenges of flight and ground project components and  instruments. To assess NOAA’s efforts to address key cost risks, we  compared program-reported data on development costs and reserves to  best practices in reserve funding as identified by the program’s  management control plan, which, in turn, reflects National Aeronautics  and Space Administration requirements. We calculated reserve  percentages using program office data on development costs and  reserves, and compared these calculations to the reserve percentages  reported by the program to management. To assess NOAA’s efforts to  address key schedule risks, we compared schedules for two key GOES-R  components to best practices in schedule development as identified in  our Cost Estimating and Assessment Guide. Similar to our previous  report, we used a five-part rating system. We then compared our previous  assessment to our current assessment to identify practices that were  improved, stayed the same, or became weaker over time. We conducted  interviews with GOES-R program staff to better understand milestone  time frames, to discuss current status and recent development challenges  for work currently being performed on GOES-R, and to understand how  the program reports costs and reserve totals. We also examined the  reliability of data on cost reserves and program schedules. Regarding  cost reserves, we examined reliability by recalculating reserve  percentages based on supporting data over a period of one year, and  compared the results to those presented by the program to management.  Regarding schedules, we created a template that examined each  schedule in areas such as missing logic, tasks completed out of  sequence, and completed tasks with start or finish dates in the future. As  a result, we found both the reserve information and the schedules to be  reliable for the purposes of conducting our analyses.
To evaluate the program’s efforts to manage changes in requirements,  we compared GOES-R practices for managing requirements changes  against best practices, which we drew from several leading industry  sources including the Software Engineering Institute’s Capability Maturity  Model®–Integration, the Program Management Institute’s Program  Manager’s Body of Knowledge, the Federal Information Security Controls  Audit Manual and the Information Technology Governance Institute’s  Control Objectives for Information and related Technology governance  framework. We assessed GOES-R practices as having satisfied, partially  satisfied, or not satisfied each best practice. We analyzed changes from  2007 to the present in the program’s Level I Requirements Document to  determine the extent of the changes. We also identified concerns about  these changes from a subset of satellite data users. We selected users  from both inside and outside NOAA’s National Weather Service, the main  GOES satellite user, based on several factors: the importance of GOES  data to the organization’s core mission, the user’s reliance on GOES  products that have changed or may change, and—for agencies outside of  NOAA—the percentage of spending devoted to meteorological  operations. The user organizations outside of NOAA included in our  review were: the US Department of Agriculture, the Department of  Transportation’s Federal Aviation Administration, and the Department of  Defense’s Navy and Air Force. User organizations inside of NOAA’s  National Weather Service included the Aviation Weather Center, Space  Weather Prediction Center, Storm Prediction Center, Environmental  Modeling Center, and a Weather Forecast Office.
To evaluate the adequacy of GOES-R contingency plans, we compared  contingency plans and procedures for both GOES satellites and the  GOES ground system against best practices developed from leading  industry sources such as the National Institute of Standards and  Technology, the Software Engineering Institute’s Capability Maturity  Model®–Integration, and our prior work. We analyzed the contingency  plans to identify strategies for various failure scenarios and determined  whether the satellite and ground system contingency plans fully  implemented, partially implemented, or did not implement each of the  practices. We also interviewed selected satellite data users to better  determine the impact of a GOES failure scenario on their operations, and  the level of communication they have had with NOAA satellite offices on  current contingency plans.
We performed our work at NOAA, National Aeronautics and Space  Administration, and US Department of Agriculture offices in the  Washington, D.C., area and at National Weather Service offices in  Kansas City, Missouri; Norman, Oklahoma; and Sterling, Virginia. We  conducted this performance audit from October 2012 to September 2013,  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Commerce
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, individuals making contributions  to this report included Colleen Phillips (assistant director), Paula Moore  (assistant director), Shaun Byrnes, Kathleen Feild, Nancy Glover,  Franklin Jackson, Kaelin Kuhn, Jason Lee, Scott Pettis, Meredith  Raymond, Maria Stattel, and Jessica Waselkow.