Polar Weather Satellites: NOAA Identified Ways to Mitigate Data Gaps, but Contingency Plans and Schedules Require Further Attention
Released: 2013-09-19T00:00:00

Letter

Background

Since the 1960s, the United States has operated two separate operational polar-orbiting meteorological satellite systems: the Polar- orbiting Operational Environmental Satellite (POES) series, which is managed by NOAA, and the Defense Meteorological Satellite Program (DMSP), which is managed by the Air Force. These satellites obtain environmental data that are processed to provide graphical weather images and specialized weather products. These satellite data are also the predominant input to numerical weather prediction models, which are a primary tool for forecasting weather days in advance—including forecasting the path and intensity of hurricanes. The weather products and models are used to predict the potential impact of severe weather so that communities and emergency managers can help prevent and mitigate its effects. Polar satellites also provide data used to monitor environmental phenomena, such as ozone depletion and drought conditions, as well as data sets that are used by researchers for a variety of studies such as climate monitoring.

Unlike geostationary satellites, which maintain a fixed position relative to the earth, polar-orbiting satellites constantly circle the earth in an almost north-south orbit, providing global coverage of conditions that affect the weather and climate. Each satellite makes about 14 orbits a day. As the earth rotates beneath it, each satellite views the entire earth’s surface twice a day. Currently, there is one operational POES satellite and two operational DMSP satellites that are positioned so that they cross the equator in the early morning, midmorning, and early afternoon. In addition, the government relies on a European satellite, called the Meteorological Operational (MetOp) satellite, for satellite observations in the midmorning orbit. In addition to the operational satellites, NOAA, the Air Force, and a European weather satellite organization maintain older satellites that still collect some data and are available to provide limited backup to the operational satellites should they degrade or fail. The last POES satellite was launched in February 2009. The Air Force plans to launch its two remaining DMSP satellites as needed. Figure 1 illustrates the current operational polar satellite constellation.

Polar satellites gather a broad range of data that are transformed into a variety of products. Satellite sensors observe different bands of radiation wavelengths, called channels, which are used for remotely determining information about the earth’s atmosphere, land surface, oceans, and the space environment. When first received, satellite data are considered raw data. To make them usable, processing centers format the data so that they are time-sequenced and include earth-location and calibration information. After formatting, these data are called raw data records. The centers further process these raw data records into channel-specific data sets, called sensor data records and temperature data records. These data records are then used to derive weather and climate products called environmental data records. These environmental data records include a wide range of atmospheric products detailing cloud coverage, temperature, humidity, and ozone distribution; land surface products showing snow cover, vegetation, and land use; ocean products depicting sea surface temperatures, sea ice, and wave height; and characterizations of the space environment. Combinations of these data records (raw, sensor, temperature, and environmental data records) are also used to derive more sophisticated products, including outputs from numerical weather models and assessments of climate trends. Figure 2 is a simplified depiction of the various stages of satellite data processing, and figure 3 depicts examples of two different weather products.

With the expectation that combining the POES and DMSP programs would reduce duplication and result in sizable cost savings, a May 1994 Presidential Decision Directive required NOAA and DOD to converge the two satellite programs into a single satellite program—NPOESS—capable of satisfying both civilian and military requirements. The converged program, NPOESS, was considered critical to the nation’s ability to maintain the continuity of data required for weather forecasting and global climate monitoring. NPOESS satellites were expected to replace the POES and DMSP satellites in the morning, midmorning, and afternoon orbits when they neared the end of their expected life spans.

To manage this program, DOD, NOAA, and NASA formed a tri-agency Integrated Program Office, with NOAA responsible for overall program management for the converged system and for satellite operations, the Air Force responsible for acquisition, and NASA responsible for facilitating the development and incorporation of new technologies into the converged system.

When the primary NPOESS contract was awarded in August 2002, the program was estimated to cost about $7 billion through 2018. The program was to include the procurement and launch of 6 satellites over the life of the program, with each satellite hosting a subset of 13 instruments. The planned instruments included 11 environmental sensors, and two systems supporting specific user services (see table 1). To reduce the risk involved in developing new technologies and to maintain climate data continuity, the program planned to launch the demonstration satellite in May 2006. This satellite was intended to demonstrate the functionality of selected instruments that would later be included on the NPOESS satellites. The first NPOESS satellite was to be available for launch in March 2008.

In the years after the program was initiated, NPOESS encountered significant technical challenges in sensor development, program cost growth, and schedule delays. By November 2005, we estimated that the program’s cost had grown to $10 billion, and the schedule for the first launch was delayed by almost 2 years. These issues led to a 2006 decision to restructure the program, which reduced the program’s functionality by decreasing the number of planned satellites from 6 to 4, and the number of instruments from 13 to 9. As part of the decision, officials decided to reduce the number of orbits from three (early morning, midmorning, and afternoon) to two (early morning and afternoon) and to rely solely on the European satellites for midmorning orbit data.

Even after the restructuring, however, the program continued to encounter technical issues in developing two sensors, significant tri- agency management challenges, schedule delays, and further cost increases. Because the schedule delays could lead to satellite data gaps, in March 2009 agency executives decided to use S-NPP as an operational satellite. Later, in August 2009, faced with costs that were expected to reach about $15 billion and launch schedules that were delayed by over 5 years, the Executive Office of the President formed a task force, led by the Office of Science and Technology Policy, to investigate the management and acquisition options that would improve the NPOESS program. As a result of this review, in February 2010, the Director of the Office of Science and Technology Policy announced that NOAA and DOD would no longer jointly procure the NPOESS satellite system; instead each agency would plan and acquire its own satellite system. Specifically, NOAA would be responsible for the afternoon orbit and the observations planned for the first and third satellites. DOD would be responsible for the early morning orbit and the observations planned for the second and fourth satellites. The partnership with the European satellite agencies for the midmorning orbit was to continue as planned. When this decision was announced, NOAA and NASA immediately began planning for a new satellite program in the afternoon orbit called JPSS. DOD began planning for a new satellite program in the morning orbit, called the Defense Weather Satellite System, but later decided to terminate the program and reassess its requirements, as directed by Congress.

Overview of Initial NOAA Plans for the JPSS Program

After the decision was made to disband the NPOESS program in 2010, NOAA began the JPSS satellite program. Key plans included: acquiring and launching two satellites for the afternoon orbit, called relying on NASA for system acquisition, engineering, and integration; completing, launching, and supporting S-NPP; developing and integrating five sensors on the two satellites; finding alternative host satellites for selected instruments that would not be accommodated on the JPSS satellites; and providing ground system support for S-NPP, JPSS, and the Defense Weather Satellite System; data communications for MetOp and DMSP; and data processing for NOAA’s use of microwave data from an international satellite.

In 2010, NOAA estimated that the life cycle costs of the JPSS program would be approximately $11.9 billion for a program lasting through fiscal year 2024, which included $2.9 billion in NOAA funds spent on NPOESS through fiscal year 2010. Subsequently, the agency undertook a cost estimating exercise where it validated that the cost of the full set of JPSS functions from fiscal year 2012 through fiscal year 2028 would be $11.3 billion. After adding the agency’s sunk costs, which had increased to $3.3 billion through fiscal year 2011, the program’s life cycle cost estimate totaled $14.6 billion. This amount was $2.7 billion higher than the $11.9 billion estimate for JPSS when NPOESS was disbanded in 2010.

In working with the Office of Management and Budget to establish the president’s fiscal year 2013 budget request, NOAA officials stated that they agreed to cap the JPSS life cycle cost at $12.9 billion through 2028, to fund JPSS at roughly $900 million per year through 2017, and to merge funding for two climate sensors into the JPSS budget. Because this cap was $1.7 billion below the expected $14.6 billion life cycle cost of the full program, NOAA decided to remove selected elements from the satellite program. Table 2 compares the planned cost, schedule, and scope of NOAA’s satellite programs at different points in time.

We have issued a series of reports on the NPOESS and JPSS programs highlighting technical issues, cost growth, and key management challenges affecting the tri-agency program structure. In June 2012, we reported that while NOAA officials communicated publicly and often about the risk of a polar satellite data gap, the agency had not established plans to mitigate the gap. At the time, NOAA officials stated that the agency would continue to use existing satellites as long as they provide data and that there were no viable alternatives to the JPSS program. However, our report noted that a more comprehensive mitigation plan was essential since it is possible that other governmental, commercial, or foreign satellites could supplement the polar satellite data. Because it could take time to adapt ground systems to receive, process, and disseminate an alternative satellite’s data, we noted that any delays in establishing mitigation plans could leave the agency little time to leverage its alternatives. We recommended that NOAA establish mitigation plans for risks associated with pending satellite gaps in the afternoon orbit as well as potential gaps in the early morning and midmorning orbits. NOAA agreed with the report’s recommendation and noted that the National Environmental Satellite, Data, and Information Service—a NOAA component agency—had performed analyses on how to mitigate potential gaps in satellite data and planned to provide a report by August 2012.

More recently, in February 2013, we added the potential gap in weather satellite data to our biennial High-Risk list. In that report, we noted that satellite data gaps in the morning or afternoon polar orbits would lead to less accurate and timely weather forecasting; as a result, advanced warning of extreme events would be affected. Such extreme events could include hurricanes, storm surges, and floods. For example, the National Weather Service performed case studies to demonstrate how its forecasts would have been affected if there were no polar satellite data in the afternoon orbit, and noted that its forecasts for the “Snowmaggedon” winter storm that hit the Mid-Atlantic coast in February 2010 would have predicted a less intense storm further east, with about half of the precipitation at 3, 4, and 5 days before the event. Specifically, the models would have under-forecasted the amount of snow by at least 10 inches. Similarly, a European weather organization recently reported that NOAA’s forecasts of Hurricane Sandy’s track could have been hundreds of miles off without polar-orbiting satellites—rather than identifying the New Jersey landfall within 30 miles 4 days before landfall, the models would have shown the storm remaining at sea. Such degradation in forecasts and warnings would place lives, property, and our nation’s critical infrastructure in danger.

We reported that the length of an afternoon polar satellite data gap could span from 17 months to 3 years or more. In one scenario, S-NPP would last its full expected 5-year life (to October 2016), and JPSS-1 would launch as soon as possible (in March 2017) and undergo on-orbit checkout for a year (until March 2018). In that case, the data gap would extend 17 months. In another scenario, S-NPP would last only 3 years as noted by NASA managers concerned with the workmanship of selected S-NPP sensors. Assuming that the JPSS-1 launch occurred in March 2017 and the satellite data were certified for official use by March 2018, this gap would extend for 41 months. Of course, any problems with JPSS- 1 development could delay the launch date and extend the gap period. Figure 4 depicts four possible gap scenarios.

We also noted that NOAA had recently established a mitigation plan for a potential 14- to 18-month gap in the afternoon orbit, which identified and prioritized options for obtaining critical observations, including alternative satellite data sources and improvements to data assimilation in models and listed technical, programmatic, and management steps needed to implement these options. However, these plans were only a beginning. We suggested that NOAA must make difficult decisions on which steps it would implement to ensure that its mitigation plans are viable when needed, including how these plans would be integrated with the agency’s broader end-to-end plans for sustaining weather forecasting capabilities.

NOAA Has Made Progress on JPSS Development, but Continues to Face Challenges in Completing S-NPP Products, Revising the Program’s Scope, and Meeting Schedules

NOAA has made progress towards JPSS program objectives of sustaining the continuity of NOAA’s polar-orbiting satellite capabilities through the S-NPP, JPSS-1, and JPSS-2 satellites by (1) delivering S-NPP data to weather forecasters and (2) by completing significant instrument and spacecraft development for the JPSS-1 satellite. However, the program is behind schedule in validating the readiness of S-NPP products and has experienced delays on the ground system schedules for the JPSS-1 satellite. Moreover, the program is moving to revise its scope and objectives to reduce costs and prioritize NOAA’s weather mission. Until it addresses challenges in product and ground system development, the program office may continue to experience delays in delivering actionable S-NPP data to users and in meeting program development schedules.

Weather Forecasters Are Using Selected S-NPP Products, but the JPSS Program Is Behind Schedule in Validating Products and Unaware of the Full Extent to Which They Are Being Used

In order to sustain polar-orbiting earth observation capabilities through the S-NPP satellite, over the past 18 months the JPSS program had planned to complete activation and commissioning of the S-NPP satellite, transition the satellite from interim to routine operations, and deliver 76 data products that were precise enough for use in operational weather observations and forecasts. To develop the precise data products, NOAA established a process for calibrating and validating its products. Under this process, most products (which are primarily sensor data records and environmental data records) proceed through three different levels of algorithm maturity—the beta, provisional, and validated levels. NOAA had originally planned to complete efforts to validate S-NPP products by October 2013, which was 2 years after the S-NPP satellite was launched. It is not enough, however, to simply deliver validated products. Both the Software Engineering Institute and GAO recommend tracking whether customers are receiving the expected value from products once they are deployed, and whether corrective actions are needed. Moreover, in April 2013 the Executive Office of the President’s National Science and Technology Council released a national strategy for civil earth observations that called for agencies to, among other things, track the extent to which earth observation data are actually being used, track whether the data had an impact, and provide data users a mechanism to provide feedback regarding ease of use, suspected quality issues, and other aspects of the data.

The JPSS program has made progress on S-NPP since launching the satellite in October 2011. Specifically, the program completed satellite activation and commissioning in March 2012, and transitioned from interim operations under NASA to routine operations under NOAA in February 2013. The program also made key upgrades to the ground system supporting S-NPP. For example, in November 2012 the office completed an interim backup command and control facility that could protect the health and safety of the satellite if unexpected issues occurred at the primary mission operations facility. In addition, the JPSS program office has been working to calibrate and validate S-NPP products in order to make them precise enough for use in weather-related operations.

While the program office plans to have 18 products validated for operational use by September 2013, it is behind schedule for the other products. Specifically, the program expects to complete validating 35 S-NPP products by the end of September 2014 and 1 other product by the end of September 2015, almost 1 and 2 years later than originally planned. In addition, the program office reported that 15 products do not need to be validated, one product’s validation date has not been established, and 6 products do not have estimated validation dates because the program plans to remove them from its requirements. The program categorized its products by their priority, ranging from priority-1 for the highest priority products, to priority-4 for the lowest priority products. According to NOAA and NASA officials, the S-NPP products’ validation has been delayed in part because of issues initially identified on VIIRS that had to be corrected and additional time needed to validate environmental data record products that require observations of seasonal weather phenomena. Further, program officials stated that they rebaselined the planned product validation timelines in November 2011 and have been generally meeting the target dates of this revised plan. Table 3 illustrates program-reported data on the number of products in each priority level, examples of products, and the estimated validation date for the last product at each level.

Even though S-NPP products are not at the validated stage in which products are ready for operational use, the National Weather Service (NWS) has accepted certain products for use in its operational systems. For example, the JPSS program office reported that NWS is using ATMS temperature data records in its operational forecasts, and that the Alaska Weather Forecast Offices are using VIIRS imagery in its forecasts. In addition, NWS’s National Centers for Environmental Prediction is evaluating CrIS sensor data records for use in numerical weather prediction, but has not yet used the data operationally because it is in the midst of a computer upgrade. Officials also stated that the program obtains information on the operational use of S-NPP data from other NOAA offices, including the National Ocean Service and the National Marine and Fisheries Service.

While NOAA is aware of these uses, it does not track the extent to which key satellite data users—including users from the Air Force, Navy, Forest Service, European weather offices, and academic institutions—have incorporated S-NPP data into their operations or if corrective actions are needed to make the products more accurate or more effective for the specific users. Program officials noted that they are not required to tailor products to meet non-NOAA user requirements, and that they do not have a tracking mechanism that would allow them to identify which entities are using the data. They noted, however, that the program obtains informal reports from customer representatives through various working groups and forums, such as the Low-earth Orbiting Requirements Working Group and the JPSS Customer Forum. While these efforts obtain information from known customer groups, they do not meet best practices for actively tracking whether customers are using the products, receiving the expected value, or in need of product corrections. Until the program office tracks the use of S-NPP and future JPSS products, it will not have full knowledge of the extent to which products are being used to assess the value they provide to end users and whether corrective actions are needed. More significantly, without information on who is using S-NPP data, NOAA will be unable to ensure that the significant investment made on this satellite is not wasted.

Development of JPSS Flight Project Is on Track, but Scheduling Issues on the Ground System Have Caused Delays

In order to sustain polar-orbiting earth observation capabilities, the program is working to complete development of the JPSS-1 systems in preparation for a March 2017 launch date. To manage this initiative, the program office organized its responsibilities into two separate projects: (1) the flight project, which includes sensors, spacecraft, and launch vehicles and (2) the ground project, which includes ground-based data processing and command and control systems. Table 4 shows the JPSS projects and their key components.

JPSS projects and components are at various stages of system development. The flight project has nearly completed instrument hardware development for the JPSS-1 satellite and has begun testing certain instruments. Also, the flight project completed a major design review for the JPSS-1 satellite’s spacecraft. While the flight project’s development is on track, the ground project experienced delays in its planned schedule that could further delay major program milestones, including key reviews required to establish the program’s cost and schedule baseline.

The flight project is generally on track with respect to planned JPSS-1 instrument and spacecraft development efforts. According to program reports of instrument development, the instruments for the JPSS-1 satellite are nearly complete. Specifically, as of July 2013, the instrument hardware ranged from 80 to 100 percent complete. Also, all of the instruments have completed or are scheduled to complete environmental testing reviews in 2013 and are to be delivered to the spacecraft by 2014. The spacecraft completed its critical design review—which evaluates whether the design is appropriately mature to continue with the final design and fabrication—in January 2013.

While individual instruments have experienced delays, the key testing milestones and delivery dates for the instruments and spacecraft have generally held constant since the last key decision point in July 2012. CERES experienced a 10-month slip in its delivery date due to a technical issue with the instrument’s internal calibration monitor, and ATMS experienced an 8-month slip to its pre-environmental review due to an issue in one of the sensor’s channels, but even accounting for these slips, the instruments have a schedule reserve of 14 and 10 months, respectively. VIIRS is expected to be the last instrument to be delivered to the spacecraft and has a schedule reserve of 6 months. Also, between July 2012 and December 2012 instrument contractors’ estimated costs at completion increased by $29 million for ATMS, CrIS, and OMPS, while the cost for VIIRS decreased by $46 million. In addition, based on program reports of technical performance, the instruments and the spacecraft are generally meeting expected technical performance. Table 5 describes the current status of the components of the JPSS-1 flight project.

The JPSS ground project has made progress in developing the ground system components, but scheduling issues have caused delays in the deployment of system upgrades. Specifically, between August 2012 and February 2013, the program office defined the ground system’s technical performance baseline, ordered and received the first increment of hardware for the next major software release, and transitioned S-NPP operational management from the JPSS program to NOAA’s office responsible for satellite operations.

However, the program has delayed the delivery of key ground system upgrades needed to support JPSS-1 because the facilities needed for hardware installation, software development, and testing activities were not available when needed. The ground system upgrades, called block 1.5 and 2.0, were originally scheduled to be delivered in January and December 2015, respectively. To address the problem in scheduling the facilities, NOAA delayed the delivery of block 1.5 and merged it with block 2.0. The program is now expecting to deliver both upgrades in December 2015. We have previously reported that compressing system development schedules introduces program risk because it implies the need to accomplish a larger number of activities in parallel and on time before the next major event can occur as planned. As a result, any complications in the merged ground system upgrades could affect the system’s readiness to support the JPSS-1 launch date.

NOAA Revised Program Scope to Focus on Weather Priorities and Reduce Costs

While NOAA is moving forward to complete product development on the S-NPP satellite and system development on the JPSS-1 satellite, the agency recently made major revisions to the program’s scope and planned capabilities and is moving to implement other scope changes as it finalizes its plans pending congressional approval. We previously reported that, as part of its fiscal year 2013 budget process, NOAA was considering removing selected elements of the program in order to reduce total program costs from $14.6 billion to $12.9 billion. By October 2012, NOAA made the following changes in the program’s scope: develop two (instead of three) TSIS instruments as well as two free- flyer spacecraft and launch vehicles to accommodate the instruments; reduce the previously planned network of fifteen ground-based receptor stations to two receptor sites at the north pole and two sites at the south pole; increase the time it takes to obtain satellite data and deliver it to the end user from 30 minutes to 80 minutes on the JPSS-2 satellite; not install an interface data processing segment at the two Navy locations or at the Air Force Weather Agency; and withdraw future support for ground operations for DOD’s Defense Weather Satellite System, which was subsequently cancelled.

More recently, as proposed by the administration, NOAA began implementing additional changes in the program’s scope and objectives in order to meet the agency’s highest-priority needs for weather forecasting and reduce program costs from $12.9 billion to $11.3 billion. Specifically, NOAA has begun to:

Transfer requirements for building the OMPS-limb and CERES follow- on climate sensors for the JPSS-2 satellite to NASA.

Transfer the first free-flyer mission from the JPSS program to a separate NOAA program, called the Polar Free Flyer program, and cancel the second free-flyer mission. More information on the Polar Free Flyer program is provided in appendix II.

Eliminate requirements for a legacy type of broadcast transmitter, which, according to NOAA officials, is in a spectrum range being crowded out by terrestrial users and is consistent with its European partners’ plans.

Reduce science and algorithm requirements for lower-priority data products.

Reduce operations and sustainment costs based on increased efficiencies through moving from customized components to more off- the-shelf solutions.

Reduce the mission life cycle by 3 years from 2028 to 2025.

While we were unable to precisely itemize the reductions in costs associated with various program changes, program officials provided rough estimates. The following table summarizes the reported cost reductions associated with key changes to the JPSS program.

While there are a number of reasons for individual changes in the program, the key reason for the June 2012 changes was to meet the program’s $12.9 billion cost cap. The reasons for the more recent changes were to reduce mission costs and complexity, focus JPSS priorities on NOAA’s weather forecasting mission, and identify opportunities to reduce potential gaps between JPSS satellites, all of which an independent study on NOAA’s satellite program recommended in July 2012.

While these are worthy goals, the changes NOAA implemented over the last 2 years will have an impact on those who rely on polar satellite data. Specifically, satellite data products will be delivered more slowly than anticipated because of the reduction in the number of ground stations, and military users may not obtain the variety of products once anticipated at the rates anticipated because of the removal of their ground-based processing subsystems. Further, while not as obvious, the impact of other changes, including the removal of the communications downlink and the reduction of requirements for certain algorithms, could also affect specific groups of satellite data users. As NOAA moves to implement these program changes, it will be important to assess and understand the impact the changes will have on satellite data users.

JPSS Schedules Demonstrate Multiple Best Scheduling Practices, but Integration Problems and Other Weaknesses Reduce Confidence in the JPSS-1 Launch Date

The JPSS program office has established a preliminary integrated master schedule and implemented multiple scheduling best practices, but the integrated master schedule is not complete and weaknesses in component schedules significantly reduce the program’s schedule quality as well as management’s ability to monitor, manage, and forecast satellite launch dates. The incomplete integrated master schedule and shortfalls in component schedules are due in part to the program’s plans to further refine the schedule as well as schedule management and reporting requirements that varied among contractors. Further, while the program is reporting a 70 percent confidence level in the JPSS-1 launch date, its analysis is likely to be overly optimistic because it was not conducted with an integrated schedule and included a component schedule with weaknesses. Until the program office completes its integrated master schedule and addresses weaknesses in component schedules, it will lack the information it needs to effectively monitor development progress, manage dependencies between schedules, and forecast the JPSS-1 satellite’s completion and launch.

The JPSS Program Has Not Yet Established a Complete Integrated Master Schedule

According to our guidance on best practices in scheduling, the success of a program depends in part on having an integrated and reliable master schedule that defines when and how long work will occur and how each activity is related to the others. The program schedule provides not only a road map for systematic project execution but also the means by which to gauge progress, identify and resolve potential problems, and promote accountability at all levels of the program. An integrated master schedule constitutes a program schedule as a network of logically linked sequences of activities that includes the entire required scope of effort, including the effort necessary from the government, contractors, and other key parties for a program’s successful execution from start to finish. Although the integrated master schedule includes all government, contractor, and external effort, the government program management office is ultimately responsible for its development and maintenance.

The JPSS program office provided a preliminary integrated master schedule in June 2013, but this schedule is incomplete. The program’s June 2013 schedule is its first attempt to document a programwide integrated master schedule since it began in October 2010. The schedule contains the scope of work for key program components, such as the JPSS-1 and JPSS-2 satellites and the ground system, and cites linkages to more detailed component schedules. However, significant weaknesses exist in the program’s schedule. Specifically, about one-third of the schedule is missing logical relationships called dependencies that are needed to depict the sequence in which activities occur. Because a logic relationship dictates the effect of an on-time, delayed, or accelerated activity on subsequent activities, any missing or incorrect logic relationship is potentially damaging to the entire network. Complete network logic between all activities is essential if the schedule is to correctly forecast the start and end dates of activities within the plan. Program documentation acknowledges that this schedule is not yet complete and the program office plans to refine it over time. Until the program office completes its integrated schedule and includes logically linked sequences of activities, it will lack the information it needs to effectively monitor development progress, manage dependencies, and forecast the JPSS-1 satellite’s completion and launch.

The Quality of JPSS-1 Component Schedules Is Inconsistent

Our scheduling guidance identifies ten best practices that support four characteristics of a high-quality, reliable schedule—comprehensive, well- constructed, credible, and controlled. A comprehensive schedule includes all government and contractor activities, reflects resources (labor, materials, and overhead) needed to do the work, and realistically reflects how long each activity will take. A well-constructed schedule includes activities that are sequenced with the most straightforward logic possible, a critical path that represents a true model of the activities that drive the project’s earliest completion date, and total float that accurately depicts schedule flexibility. A credible schedule reflects the order of events necessary to achieve aggregated products or outcomes (horizontal traceability) and maps varying levels of the schedule to one another (vertical traceability). Also, a credible schedule includes data about risks and opportunities that are used to predict a level of confidence in meeting the project’s completion date. A controlled schedule is updated periodically by trained schedulers using actual progress and logic to realistically forecast dates for program activities and is compared against a designated baseline schedule to measure, monitor, and report the project’s progress. The JPSS program office is applying NASA’s schedule management handbook guidance to manage its schedules, which is largely consistent with our guidance on scheduling best practices. Table 7 provides more detail on the best practices and key characteristics of a reliable schedule.

The quality of three selected component schedules supporting the JPSS- 1 mission—VIIRS, the spacecraft, and the ground system—was inconsistent with respect to implementing the characteristics of a high- quality, reliable schedule. Each schedule had strengths and weaknesses with respect to sound scheduling practices, but VIIRS was a stronger schedule with fewer weaknesses compared to the ground system and spacecraft schedules. Since the reliability of an integrated schedule depends in part on the reliability of its subordinate schedules, schedule quality weaknesses in these schedules could transfer to an IMS derived from them. Table 8 identifies the quality of each of the selected JPSS-1 component schedules based on the extent to which they met ten best practices of high-quality and reliable schedules; the discussion that follows highlights observed strengths and weaknesses from each schedule. In addition, appendix III includes a more detailed assessment of each schedule against the ten best practices.

Of the ten best practices, the ground system schedule minimally met two best practices, partially met two best practices, and substantially met six best practices. There were strengths in the ground schedule in that the contractor established a clear process for integrated information between the schedule and its resource management software and the contractor has performed resource leveling on the schedule. In addition, the contractor stated that people responsible for the activities estimated activity durations. Also, the contractor stated that it performs wellness checks on the quality of the schedule after each update to identify issues associated with missing logic or date constraints and provides a monthly status briefing to the JPSS program office that addresses the status of external schedule handoffs.

However, there were also weaknesses in the ground schedule. For example, activities on the critical path with date constraints are preventing accurate calculations of the schedule’s total float, or flexibility. In order for the critical path to be valid, the activities on the critical path must also have reasonable total float. Without a critical path that accurately calculates schedule flexibility, the program office will not be able to provide reliable timeline estimates or identify when problems or changes may occur and their effect on downstream work. Moreover, while the contractor conducted a schedule risk analysis on the schedule, that analysis was for select near-term milestones rather than the readiness of the ground system for the launch of JPSS-1 and it did not include the risks most likely to delay the project. A schedule risk analysis should be conducted through the finish milestone and should include risk data to determine activities that most often end up on the critical path.

Of the ten best practices, the spacecraft schedule partially met eight best practices, and substantially met two best practices. There were strengths in the spacecraft schedule in that it was horizontally and vertically traceable; the contractor provided evidence of monthly progress updates to management, including status reporting of key milestones, handoffs, explanations of date changes, and an analysis of the critical and near- critical paths; the contractor conducted a schedule risk analysis; and the schedule included baseline dates of activities for comparisons of actual performance to date.

However, there were also weaknesses in the spacecraft schedule. For example, the schedule had a low level of detail and included one-third of remaining activities with durations greater than 44 days, even after accounting for undefined and procurement-related activities. When establishing the durations of activities, they should be reasonably short and meaningful and allow for discrete progress measurement. Durations longer than 2 months do not facilitate objective measurement of accomplished effort and the milestone to detail activity ratio does not allow for effective progress measurement and reporting. As another example of a quality shortfall, the schedule was overly flexible with high float values that were not justified in schedule documentation. Specifically, 70 percent of remaining activities had about 5 business weeks of float, including 67 activities that had over 1,000 days of float, meaning that these activities could slip approximately 3.5 years without affecting the project’s completion date. In order to establish reasonable total float, there should be documented justification for high float values in the schedule. Without this, it is unclear whether float values are high due to factors accepted by management and which are due to incomplete logic or other issues.

The VIIRS schedule partially met one best practice, substantially met seven best practices, and fully met two best practices. There were strengths in the VIIRS schedule in that the contractor established a clear process for integrating information between the schedule and resource management software, stated that durations were estimated by the people responsible for the activities based on work to be done, and justified in its schedule documentation activities with durations longer than 44 days. In addition, the contractor justified in schedule documentation the use of all date constraints, identified a valid driving path of activities for managing the program, and identified reasonable float values or justified them to the JPSS program office. Further, the contractor provided a schedule narrative accompanying each status update, which describes the status of key milestone dates (including the program finish date); explanations for changes in key dates; and a description of critical paths.

However, there were also weaknesses in the VIIRS schedule. For example, the schedule had milestones that represented handoffs between contractor integrated product teams, but it did not include handoffs to the JPSS program office. In order to verify a schedule’s horizontal traceability, handoffs should link products and outcomes associated with other sequenced activities. Without this, there could be different expectations between management and activity owners. As another example, the contractor conducted a schedule risk analysis with a good schedule network and obtained three different duration estimates from subject matter experts. However, the duration estimates did not reflect risks from the project’s risk register and the analysis was focused only on activities on the critical path. This approach is flawed because activities that are not currently on the critical path could become critical as risks occur.

The inconsistency in quality among the three schedules has multiple causes. Program and contractor officials explained that certain weaknesses have been corrected with updated schedules. In other cases, the weaknesses lacked documented explanation in part because the JPSS program office did not require contractors to provide such documentation. Based on program schedule documentation, the schedule management and reporting requirements varied across contractors without documented justification for tailored approaches, which may partially explain the inconsistency in practices among the schedules. Since the reliability of an integrated schedule depends in part on the reliability of its subordinate schedules, schedule quality weaknesses in these schedules will transfer to an integrated master schedule derived from them. Consequently, the extent to which there are quality weaknesses in JPSS-1 support schedules further constrains the program’s ability to monitor progress, manage key dependencies, and forecast completion dates. Until the program office addresses the scheduling shortfalls in its component schedules, the JPSS schedule will have lower quality and reduced reliability as a management tool for monitoring and forecasting satellite launch dates.

Program Has Confidence in the JPSS-1 Schedule, but Its Assumptions Do Not Reflect Weaknesses in the Underlying Data

According to our guidance on best practices in scheduling, a schedule risk analysis uses statistical techniques to predict a level of confidence in meeting a program’s completion date. This analysis focuses on key risks and how they affect the schedule’s activities. The analysis does not focus solely on the critical path because, with risk considered, any activity may potentially affect the program’s completion date. By relying on statistical simulations to randomly vary activity durations according to the probability of occurrence for certain durations and risks, the analysis seeks to develop a probability distribution of possible completion dates that reflect the program plan and enable an organization to match a date to its degree of risk tolerance.

The JPSS program office has conducted a schedule risk analysis on the JPSS-1 mission schedule (and launch date) through NASA’s joint cost and schedule confidence level (JCL) process. The JCL implemented by the JPSS program office represents a best practice in schedule management for establishing a credible schedule and reflects a robust schedule risk analysis conducted on key JPSS-1 schedule components. For example, the analysis assessed the impacts of key risks from the risk register and how multiple duration estimates for activities, based on documented uncertainty distributions, could affect the schedule. Based on the results of the JCL, the program office reports that its level of confidence in the JPSS-1 schedule is 70 percent and that it has sufficient schedule reserve to maintain a launch date of no later than March 2017.

However, the program office’s level of confidence in the JPSS-1 schedule may be overly optimistic for two key reasons. First, the model that the program office used was based on flight project activities rather than an integrated schedule consisting of flight, ground, program office, and other activities relevant to the development and launch of JPSS-1. As a result, the JPSS program office’s confidence level projections do not factor in the ongoing scheduling issues that are impacting the ground project. Had those issues been considered, the JPSS-1 confidence level would have been lower. Second, there are concerns regarding the spacecraft schedule’s quality as discussed in the previous section. Factoring in these concerns, the confidence of the JPSS-1 satellite’s schedule and projected launch date would be lower. We have previously reported that when using the JCL, NASA projects did not always include relevant cost and risk inputs.

While program officials noted that they included key ground system risks in their calculations, they did not include ground system scope in the JCL because it was too difficult to allocate ground system components to individual missions. Moreover, officials stated that they do not plan to include ground project or program office activities in future JCL updates. While it may have been difficult to include ground system scope in the JCL, without this, the program’s schedule risk analysis and JCL do not reflect the full amount of work to be performed leading to JPSS-1 launch.

Until the program office conducts a schedule risk analysis on an integrated schedule that includes the entire scope of effort and addresses quality shortfalls of relevant component schedules, it will have less assurance of meeting the planned March 2017 launch date for JPSS-1.

NOAA Has Analyzed Alternatives for Addressing Gaps in Satellite Data, but Lacks a Comprehensive Contingency Plan

While NOAA has identified multiple ways to help mitigate expected gaps in polar satellite data, it has not yet developed and implemented a comprehensive contingency plan. In October 2012, NOAA established a plan to address the impact of potential gaps in polar afternoon satellite data and contracted for a technical assessment that generated additional alternatives for the agency to consider. However, NOAA’s mitigation plan has shortfalls when compared to government and industry best practices. Moreover, NOAA intends to update its plan by fall 2013 by integrating alternatives generated from the contractor’s technical assessment. Until NOAA establishes a comprehensive contingency plan that addresses key shortfalls, it may not be positioned to effectively mitigate anticipated gaps in polar satellite coverage.

NOAA Identified Multiple Ways to Mitigate Polar Satellite Data Gaps

Polar satellites are essential to NOAA’s mission to understand and predict changes in climate, weather, oceans, and coasts. Satellite data gaps in the morning or afternoon polar orbits would lead to less accurate and timely weather forecasting; as a result, advanced warning of extreme events would be affected. In June 2012, we reported that while NOAA officials communicated publicly and often about the risk of a polar satellite data gap, the agency had not established plans to mitigate the gap. We recommended that NOAA establish mitigation plans for pending satellite gaps in the afternoon orbit as well as potential gaps in the early morning and midmorning orbits and NOAA agreed with the report’s recommendation.

In October 2012, NOAA established a mitigation plan to address the impact of potential gaps in polar afternoon satellite data. This plan identifies alternatives for mitigating the risk of a 14- to 18-month gap in the afternoon orbit beginning in March 2016, between the current polar satellite and the JPSS-1 satellite. Key alternatives include utilizing different satellites as data sources and improving data assimilation in models. The plan also lists technical, programmatic, and management actions needed to implement these options. Table 9 provides an overview of NOAA’s polar satellite gap mitigation plan.

However, NOAA did not implement the actions identified in its mitigation plan and decided to identify additional alternatives. In October 2012, at the direction of the Under Secretary of Commerce for Oceans and Atmosphere (who is also the NOAA Administrator), NOAA contracted for a detailed technical assessment of alternatives to mitigate the degradation of products caused by a gap in satellite data in the afternoon polar orbit. This assessment solicited input from experts within and outside of NOAA and resulted in the following alternatives: rely on DOD’s DMSP satellite; expand the use of radio occultation data, including funding the ground segment for a follow-on United States/Taiwan radio occultation mission; use atmospheric motion vectors (observed wind data); utilize future geostationary advanced imagery data; expand the use of aircraft observations; expand the use of targeted observations for high-impact events; implement a 4-dimensional hybrid data assimilation system (by adding a time dimension) ; improve data assimilation of cloud-impacted radiances; implement blends of global models, such as European and Canadian models; accelerate global model research to operations; sustain the use of high-latitude direct readout imagery; and rely on China’s future Feng Yun-3 satellite.

Moving forward, NOAA officials stated that they are currently considering the additional alternatives and that the agency intends to integrate a final set of alternatives into its existing mitigation plan by the fall of 2013.

NOAA Does Not Yet Have a Comprehensive Contingency Plan

Government and industry best practices call for the development of contingency plans to maintain an organization’s essential functions in the case of an adverse event. As a complement to risk mitigation, contingency planning includes strategies that attempt to reduce or control the impact of risks should they occur. These practices identified by, for example, the National Institute of Standards and Technology and the Software Engineering Institute, include key elements such as defining failure scenarios, identifying and selecting strategies to address failure scenarios, developing procedures and actions to implement the selected strategies, testing the plans, and involving affected stakeholders. These elements can be grouped into categories, including (1) identifying failure scenarios and impacts, (2) developing contingency plans, and (3) validating and implementing contingency plans (see table 10).

By documenting its mitigation plan and conducting a study on additional alternatives, NOAA has taken positive steps towards establishing a contingency plan for handling the potential impact of satellite data gaps in the afternoon polar orbit. However, NOAA does not yet have a comprehensive contingency plan because it has not yet selected the strategies to be implemented, or established procedures and actions to implement the selected strategies. In addition, there are shortfalls in the agency’s current plans as compared to government and industry best practices, such as not always identifying specific actions with defined roles and responsibilities, timelines, and triggers. Moreover, multiple steps remain in testing, validating, and implementing the contingency plan. The following table provides an assessment of the extent to which NOAA’s mitigation plan met contingency planning practices in three general categories.

NOAA officials stated that the agency is continuing to work on refinements to its gap mitigation plan, and that they anticipate issuing an updated plan in fall 2013 that will reflect additional alternatives. While NOAA expects to update its plan, the agency does not yet have a schedule for adding key elements—such as specific actions, roles and responsibilities, timelines, and triggers—for each alternative. Until NOAA establishes a comprehensive contingency plan that integrates its strategies and addresses the elements identified above to improve its plans, it may not be sufficiently prepared to mitigate potential gaps in polar satellite coverage.

Conclusions

While NOAA has made noteworthy progress over the past year in utilizing S-NPP data in weather forecasts and developing instrument and spacecraft components of the JPSS-1 satellite, the agency is facing challenges in its efforts to ensure sustained satellite observations. Specifically, NOAA does not expect to validate key S-NPP products until September 2014—nearly 3 years after the satellite’s launch. Also, the agency does not track the usage of its satellite products or obtain feedback on them, which limits the program’s ability to ensure that satellite products are useful. Further, the program experienced scheduling problems on its ground systems, which led to a delay in planned system upgrades. Until NOAA establishes a way to track which agencies are using its products and to obtain feedback on those products, the program office may continue to experience delays in delivering actionable S-NPP data to users.

Almost 3 years after the JPSS program was established, it lacks a complete integrated master schedule. While program officials recently established a preliminary integrated master schedule, the schedule lacks proper linkage among dependent activities, which limits its ability to calculate dates and predict changes in the future. Further, the quality of component schedules varied for certain practices. These issues raise questions about the program’s 70 percent joint cost and schedule confidence level in the JPSS-1 launch date. Until the program office develops a complete integrated schedule and addresses weaknesses in component schedules, it will lack the information needed to effectively monitor development progress and ensure the planned JPSS-1 launch date.

NOAA has taken steps to mitigate an anticipated gap in polar afternoon satellite data, but its efforts are incomplete. Specifically, the agency has not yet established a comprehensive contingency plan that identifies specific actions with defined roles and responsibilities, timelines, and triggers for contingency strategies. Moreover, the agency’s recent assessment of a larger set of alternatives has not yet been integrated with its mitigation plans. As a result, the agency faces important decisions as to whether and how the various alternatives should be carried out. While NOAA plans to add alternatives to its mitigation plan by fall 2013, it does not yet have plans to add the other key components. Until NOAA establishes a comprehensive contingency plan that addresses these shortfalls, its plan for mitigating potential gaps in the polar orbit may not be effective in avoiding significant impacts to NOAA’s weather mission.

Recommendations for Executive Action

Given the importance of having reliable schedules for managing JPSS satellite launch dates and the significance of polar-orbiting satellite data to weather forecasts, we recommend that the Secretary of Commerce direct the Administrator of NOAA to track the extent to which key groups of satellite data users are using S-NPP and JPSS products, and obtain feedback on these products; establish a complete JPSS program integrated master schedule that includes a logically linked sequence of activities; address the shortfalls in the ground system and spacecraft component schedules outlined in this report; after completing the integrated master schedule and addressing shortfalls in component schedules, update the joint cost and schedule confidence level for JPSS-1, if warranted and justified; establish a comprehensive contingency plan for potential satellite data gaps in the polar orbit that is consistent with contingency planning best practices identified in this report. The plan should include, for example, specific contingency actions with defined roles and responsibilities, timelines, and triggers; analysis of the impact of lost data from the morning orbits; and identification of opportunities to accelerate the calibration and validation phase of JPSS-1.

Agency Comments

We sought comments on a draft of our report from the Department of Commerce and NASA. We received written comments from Commerce transmitting NOAA’s comments. NOAA concurred with all five of our recommendations and identified steps that it is taking to implement them. It also provided technical comments, which we have incorporated into our report, as appropriate. NOAA’s comments are reprinted in appendix IV.

NASA did not provide comments on the report’s findings or recommendations, but noted that it would provide any input it might have to NOAA for inclusion in NOAA’s comments.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution of it until 30 days from the date of this letter. We are sending copies of this report to interested congressional committees, the Secretary of Commerce, the Administrator of NASA, the Director of the Office of Management and Budget, and other interested parties. In addition, this report will be available on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions on the matters discussed in this report, please contact me at (202) 512-9286 or at pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) evaluate the National Oceanic and Atmospheric Administration’s (NOAA) progress in meeting the Joint Polar Satellite System (JPSS) program’s objectives of sustaining the continuity of NOAA’s polar-orbiting satellite system through the Suomi National Polar-orbiting Partnership (S-NPP) and JPSS satellites, (2) evaluate the quality of the JPSS program schedule, and (3) assess NOAA's plans to address potential gaps in polar satellite data.

To evaluate NOAA's progress in meeting JPSS program objectives, we assessed (1) the status of activities supporting the operational S-NPP satellite, (2) progress on efforts to develop the JPSS-1 satellite, and (3) recent changes in JPSS program scope. A more detailed description of our activities in each of these areas follows.

S-NPP progress: We reviewed monthly program reports to identify the status of key upgrades to the ground system supporting S-NPP and the efforts to transition operational control of the satellite to NOAA. In addition, we compared the program’s current estimated completion dates for S-NPP products to original program estimates for when the products would be available for operational use. We compared program office information on the extent to which S-NPP products were being used to best practices in evaluating the use of completed products. We also interviewed program officials about algorithm maturity and the extent to which users are using S-NPP products.

JPSS-1 progress: We analyzed plans and reports on system development efforts for the JPSS-1 satellite. Specifically, we reviewed the JPSS-1 mission preliminary design review package to assess completion of work on the instruments, spacecraft, and ground system as well as cost, schedule, and technical performance for the JPSS-1 satellite. We also examined JPSS program office monthly status reports on system development progress to identify variances and corrective actions being taken to address the most critical issues and risks to the program. We interviewed JPSS program officials to discuss system development status. We assessed the reliability of reported milestone dates for top-level milestones by examining multiple project status reports at different points in time for consistent reporting of dates or explanations of any changes and compared reported dates to source schedule data. We determined that the milestone data were sufficiently reliable for our reporting purposes.

Changes in JPSS program scope: We compared the program’s requirements as of September 2011 to the program’s updated plans and requirements as of May 2013 to identify key changes and to assess whether changes in capabilities have impacted program goals and objectives. We interviewed program officials about changes in the JPSS program’s scope. We assessed the reliability of the program’s estimated savings from program scope changes by comparing them to program documentation on prior and current cost estimates and found that the estimates were sufficient for our purposes.

To evaluate the quality of NOAA's program schedule, we used an exposure draft of GAO’s Schedule Assessment Guide to assess schedule management practices and characteristics of selected contractor schedules. We selected and analyzed three component contractor schedules—the ground system, the spacecraft, and the Visible/Infrared Imager/Radiometer Suite instrument—because these schedules represented the critical path for flight and the entire ground system development schedule that was either already or likely to be driving the JPSS-1 satellite launch date. We also analyzed schedule metrics as a part of that analysis to highlight potential areas of strengths and weakness in, among other things, schedule logic, use of resources, task duration, float, and task completion. In order to assess each schedule against the ten best practices, we traced and verified underlying support and determined whether the program office or contractor provided a small portion, about half, a large portion, or complete evidence that satisfied the criterion and assigned a score depicting that the practices were met, minimally met, partially met, substantially met, or fully met. By examining the schedules against our guidance, we conducted a reliability assessment on each of the schedules and incorporated our findings on reliability limitations in the analysis of each component schedule. We reviewed documentation on a schedule risk assessment the JPSS program office conducted on JPSS-1 flight project schedules to identify assumptions and results of its analysis and to assess the reliability of the reported JPSS joint cost and schedule confidence level. We interviewed government and contractor officials to discuss reasons for observed shortfalls in schedule management practices. We determined that the schedules were sufficiently reliable for our reporting purposes and our report notes the instances where reliability concerns affect the quality of the schedules as well as the program’s schedule risk assessment.

To assess plans to address potential gaps in polar satellite data, we reviewed NOAA’s October 2012 polar satellite gap mitigation plan and a subsequent technical assessment as well as NOAA’s plans for implementing recommendations from the assessment. We compared elements of the plan and assessment against best practices developed from leading government and industry sources such as the National Institute of Standards and Technology, the Software Engineering Institute’s Capability Maturity Model® Integration, and our prior report. Based on that analysis, we identified shortfalls in NOAA’s current plans as well as key remaining activities for the agency to accomplish. We interviewed NOAA headquarters staff and JPSS program officials about the technical assessment and their plans.

We performed our work at NASA and NOAA offices in the Washington, D.C. area. We conducted this performance audit from October 2012 through September 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: NOAA Plans to Transfer Selected JPSS Program Components to the Polar Free Flyer Program

In order to reduce Joint Polar Satellite System (JPSS) program costs and increase the program’s focus on its weather mission, the National Oceanic and Atmospheric Administration (NOAA) plans to transfer key program components to a separate program, called the Polar Free Flyer program. After establishing JPSS in 2010, NOAA committed to developing three units of the Total and Spectral Solar Irradiance Sensor (TSIS) and to finding a spacecraft and launch accommodation for three instruments that would not be on the JPSS satellite: TSIS, the Advanced Data Collection System (A-DCS), and the Search and Rescue Satellite- Aided Tracking (SARSAT) system. As of June 2012, the JPSS program planned to launch two stand-alone satellites (called free flyers) to accommodate two suites of these instruments. However, NOAA recently made several decisions that affect these commitments, and expects to finalize these plans by the end of September 2013:

NOAA plans to transfer responsibility for developing TSIS and accommodating the launch of the three instruments out of the JPSS program and into a newly established Polar Free Flyer program. According to JPSS program officials, a transition plan for the new program is under review and selected staff positions have been filled.

The Polar Free Flyer program will deliver a single free flyer mission instead of the two missions planned under the JPSS program.

NOAA will transfer the responsibility for developing the second TSIS instrument to the National Aeronautics and Space Administration (NASA), rely on an Air Force Global Positioning System mission to continue SARSAT coverage, and find a launch vehicle to accommodate an additional A-DCS instrument.

NOAA plans to use the JPSS ground system to support the Polar Free Flyer Program.

The JPSS program plans to award a contract in fiscal year 2014 for a spacecraft that is to accommodate the TSIS, A-DCS, and SARSAT instruments. The three instruments are in development and testing, and are expected to be delivered to the satellite by 2015. The planned launch readiness date for the free-flyer mission was originally July 2016, but that date may change pending the outcome of the spacecraft contract award. Also, the program is looking to share a launch vehicle with some other mission to reduce launch costs. However, the program office is not aware of any ride-sharing opportunities that could accommodate the mission’s planned launch readiness date.

Appendix III: Assessment of JPSS Component Schedules Implementation of Best Practices in Scheduling

The following tables identify detailed assessments of the extent to which three component schedules supporting the JPSS-1 schedule met the ten best practices and four characteristics of a high-quality, reliable schedule. Table 12 provides an assessment of the ground system contractor’s schedule, which integrates activities from seven components of the ground system; table 13 provides an assessment of the spacecraft contractor’s detailed schedule; and table 14 provides an assessment of the VIIRS contractor’s detailed schedule.

The following information describes the key that we used in tables 12 through 14 to convey the results of our assessment of the schedules’ consistency with an exposure draft of GAO best practices for schedule management. ● Met: The program office or contractor provided complete evidence that satisfies the entire criterion. ◕ Substantially met: The program office or contractor provided evidence that satisfies a large portion of the criterion. ◑ Partially met: The program office or contractor provided evidence that satisfies about half of the criterion. ◔ Minimally met: The program office or contractor provided evidence that satisfies a small portion of the criterion. ○ Not met: The program office or contractor provided no evidence that satisfies any of the criterion.

Schedule characteristic or best practice Comprehensive Capturing all activities

The schedule largely reflects the statement of work. However, the schedule only partially reflects the work breakdown structure and includes 40 activities that are marked as both summary activities and milestones.

The contractor has established a clear process for integrating information between the schedule and the resource management software. However, resource leveling has been performed outside of the schedule, which limits the effectiveness of the process.

According to the contractor, durations were estimated by the people responsible for the activities based on work to be done. Additionally, calendars were used to specify valid working times for all activities. However, over 35 percent of the activities in the schedule were of long duration, and only half of these were justified in schedule documentation.

Schedule characteristic or best practice Well-constructed Sequencing all activities

A majority of the activities in the schedule had dependencies, and the schedule’s relationships were largely finish-to-start. However, program officials did not justify in schedule documentation the small number of activities with missing dependencies, date constraints, and lags.

The critical path and driving path are not fully valid because they are not free of long activities, constraints, and lags. Moreover, considering the schedule as a whole, the schedule software may not be calculating the true critical path of the project because the use of more than 800 constraints. These may result in float values that present an unrealistic view of the critical path.

According to contractor officials, float values have been assessed as part of regularly scheduled health checks and they have determined that for certain cases float values are necessarily high. However, not all float values calculated by the schedule are reasonable and many values do not accurately reflect true schedule flexibility. Additionally, the JPSS program office did not provide a documented assessment of total float values that appear to be excessive to show that the team agrees with the logic and that the float is consistent with the plan.

The schedule is vertically traceable in all but one of the milestones that we reviewed, meaning that it allows activity owners to trace activities to higher-level milestones with intermediate and summary schedules. However, the schedule is not fully horizontally traceable—that is, although the schedule includes giver/receiver milestones that are defined in the schedule documentation, the schedule was not always affected by activities whose durations were extended by hundreds of days.

The contractor conducted a schedule risk analysis with a schedule network that partially meets the characteristics associated with a good schedule network, as well three point duration estimates that were captured from control account managers. However, the analysis was conducted for select near-term milestones—not to the readiness of the ground system for the launch of JPSS-1. Additionally, the analysis did not include risks most likely to delay the project, the paths or activities that are most likely to delay the project, and the activities that most often ended up on the critical path.

Responsibility for changing the schedule has been assigned to someone who has the proper training and experience in critical path method scheduling and the schedule is free of clearly erroneous progress information. However, although the contractor provides a monthly program management briefing that addresses the status of external giver/receiver activities, it does not address the status of key milestone dates, changes in network logic, or critical paths.

A baseline schedule exists and is compared to the current schedule to track variances from the plan. According to contractor officials, a formal change control process is used to make changes to the baseline. However, the contractor’s rolling wave reports do not satisfy all elements of a baseline schedule document. A baseline schedule document is a single document that describes, among other things, the organization of the IMS; the logic of the network; the basic approach to managing resources; the schedule’s unique features; and justification for lags, date constraints, and long activity durations.

The schedule reflects the work necessary to build the spacecraft, and schedule activities are mapped to the contract data requirements list and contractor work breakdown structure numbers. The schedule contains a low level of detail, which reflects the contractor’s role as integrator for multiple vendors in a fixed-price environment. However, with a nearly 1:1 ratio of detail activities to milestones, the schedule would benefit from increased detail into work activities.

The contractor has established a clear process for integrating information between the schedule and the resource management software. However, resource leveling has been performed outside of the schedule, which limits the effectiveness of the process.

The contractor has experience in developing spacecraft similar to JPSS-1, including S-NPP. Contractor officials stated that they obtained duration estimates for activities from engineers that were responsible for them while other engineers conducted peer reviews on those estimates. However, durations in general appear too long to facilitate objective measurement of accomplished effort. Even accounting for procurement-related activities and level-of-effort type recurring meeting activities, one-third of all remaining activities are longer than 2 business months.

The schedule was partially logically sequenced. Approximately 20 percent of all remaining activities and milestones were missing predecessor links, successor links or both. Officials stated that many of these activities were related to contract data requirements list deliveries and internal or external handoffs (called givers/receivers). We found other areas of questionable sequencing logic. For instance, there are about 10 percent of remaining activities in the schedule that have lags and leads, including some instances of leads with start-to-finish logic—a particularly abnormal logical relationship. We also found date constraints pervasive throughout the schedule: 140 activities have soft constraints and 17 have hard constraints. Hard constraints are useful for calculating the amount of float available in the schedule and, therefore, the realism of the required project finish date and available resources during schedule development. However, they may be abused if they force activities to occur on specific dates that are determined off- line without much regard for the realism of the assumptions necessary to achieve them.

The schedule defines activities with zero total float as critical. However, partly because of logic issues, the critical path as calculated by the scheduling software was convoluted and most likely unreliable. The path includes lags, leads, long-duration activities, and activities with hard constraints, which by definition will appear as critical. Officials stated they agreed that software-calculated critical paths cannot be relied upon in a complex schedule, and said they report the longest (or driving path) to management. Ideally, the critical path and the longest path should be the same, but our analysis found the longest path to be somewhat different than the default critical path; it does not include several activities that appeared on the critical path because of their date constraints. In addition, the longest path also includes several near-term, nonprocurement-related activities with long durations, spanning between 84 and 365 days.

Schedule characteristic or best practice Ensuring reasonable total float

Officials stated that the total float values calculated by the schedule accurately reflect true schedule flexibility. However, we found that the schedule appears overly flexible due to high amounts of total float. 70 percent of remaining activities and milestones have greater than 30 days (about 5 business weeks) of total float. This includes 67 activities (8 percent of remaining) with over 1,000 days of float, meaning these activities can slip more than 3.5 business years before impacting the planned finish date of the project. Without documented justification for high float values in the schedule, it is not clear which are explained by milestones without successors, which are due to schedule maintenance, and which are due to incomplete logic.

The schedule is vertically traceable, with dates in the detail schedule mapping to higher- level management briefing charts. The schedule is generally horizontally traceable. The schedule clearly identifies givers and receivers and negative total float calculations respond appropriately when significant delays are introduced into the network. However, negative float is calculated because key milestones are constrained. While the negative float may be an accurate assessment of potential delay, management may not be aware of potential delays when constrained dates are reported in summary-level schedules.

Officials stated that they follow an internal process to perform schedule risk analyses on the schedule. Officials also stated that three-point durations are applied to activities, correlation is accounted for, and a Monte Carlo analysis is run on the schedule to derive probabilities for forecasted dates. Although the contractor has no contractual requirement to share schedule risk analysis results with the JPSS program office, it provided a summary of its risk assessment report and instructions. However, this summary information did not include supporting details such as risk data inputs and data normalization techniques and the contractor did not incorporate correlation or perform the schedule risk analysis on a logically sound (well-constructed) schedule.

Schedule progress is updated monthly and the schedule is delivered to the JPSS program office in accordance with contractual requirements. While a formal schedule narrative does not accompany the schedule delivery to the government, much of the narrative information—such as the status of key milestones and handoffs, explanations for changes in key dates, and an overview of critical and near-critical paths—is conveyed in monthly management meetings. However, 26 activities had start or finish dates in the past. Of these, 12 activities could be explained by obsolete scope of work. We also found 12 out-of-sequence activities, representing 13 percent of in-progress activities.

Contractor officials stated that they maintain schedule baseline information in the default baseline fields in the schedule and we found that baseline dates were set in the schedule. However, a schedule baseline document was not created for the schedule baseline. We found 104 activities in the schedule without baseline dates, 72 of which are complete or are planned to start by 2014. The majority of start variances appear reasonable, but we did find start variances ranging from -221 days (221 days ahead of schedule) to 237 days (237 days delayed). Despite the significant variances noted, it is commendable that the schedule includes baseline information that allows for analysis and monitoring of dates’ variances.

The schedule largely reflects the work breakdown structure and statement of work. However, the schedule does not reflect work to be performed by a subcontractor and includes 10 activities that are marked as both summary activities and milestones.

The contractor has established a clear process for integrating information between the schedule and the resource management software. However, resource leveling has been performed outside of the schedule, which limits the effectiveness of the process.

According to the contractor, durations were estimated by the people responsible for the activities based on work to be done, realistic assumptions about available resources, productivity, normal interferences and distractions, and reliance on others. Further, the contractor justified in its schedule documentation virtually all activities with durations longer than 44 days.

All but one activity in the schedule has at least one predecessor and one successor, and that activity was justified in the schedule documentation. Additionally, every schedule date constraint was justified in schedule documentation. However, the schedule has a very small number of activities with dangling logic. Further, although explanations were provided for most of the small number of lags, the explanations did not justify their use.

Program office and contractor officials use the driving path to manage the program, which is preferred because it represents the activities that are driving the sequence of start dates directly affecting the estimated finish date. However, the driving path and the critical path to key milestones should be the same, and they are not. Also, the critical path is not valid because it contains level of effort activities.

The program office has defined reasonable float values, and the values associated with the schedule largely fit that definition. For those float values that were not reasonable, the program office provided a documented assessment of those values to show that the team agrees with the logic and that the float is consistent with the plan. However, the schedule has a small number of activities that have unrealistic float values.

The schedule is largely horizontally traceable. In particular, the schedule is affected by activities whose durations are extended by hundreds of days, and it includes giver/receiver milestones that represent handoffs between contractor integrated project teams. However, the schedule does not include all givers/receivers between the contractor and the program office. Additionally, the schedule is vertically traceable. Specifically, it allows activity owners to trace activities to higher-level milestones with intermediate and summary schedules.

A schedule risk analysis was conducted with a good schedule network, and three point duration estimates that were captured from subject matter experts. However, the duration estimates did not reflect risks from the project’s risk register and the analysis was focused on only the deterministic critical path and near-critical path.

GAO assessment Examples of strengths and weaknesses

Responsibility for changing or updating the schedule has been assigned to someone who has the proper training and experience in critical path method scheduling. Additionally, the schedule is free of clearly erroneous progress information. Further, the contractor provides a schedule narrative accompanying each status update, which describes the status of key milestone dates (including the program finish date); explanations for changes in key dates; and a description of the critical paths.

A baseline schedule exists and is compared to the current schedule to track variances. However, the contractor did not have a baseline schedule document. A baseline schedule document is a single document that describes, among other things, the organization of the IMS; the logic of the network; the basic approach to managing resources; the schedule’s unique features; and justification for lags, date constraints, and long activity durations.

Appendix IV: Comments from the Department of Commerce

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

David A. Powner (202) 512-9286 or pownerd@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Colleen Phillips (Assistant Director), Paula Moore (Assistant Director), Shaun Byrnes, Juaná Collymore, Lynn Espedido, Kate Feild, Nancy Glover, Franklin Jackson, Kaelin Kuhn, Jason Lee, Joshua Leiling, and Maria Stattel made key contributions to this report.