Weather Forecasting: NWS Has Not Demonstrated That New Processing System Will Improve Mission Effectiveness
Released: 1996-02-29T00:00:00

Letter

Background

Since the early 1980s, NWS has been modernizing its observing, information processing, and communications systems to improve the accuracy, timeliness, and efficiency of weather forecasts and warnings. The $4.5 billion modernization includes four major systems—AWIPS, the Next Generation Geostationary Operational Environmental Satellites (GOES-Next), the Next Generation Weather Radars (NEXRAD), and the Automated Surface Observing System (ASOS). GOES-Next, NEXRAD, and ASOS form the foundation of NWS’ observing infrastructure, watching the weather from earth and space. The two orbiting GOES-Next satellites each provide as many as 1,200 digital weather images daily to each forecast office, compared to less than 100 from their predecessors. The almost 150 NEXRADs are to blanket the continental United States and using Doppler radar technology, allow forecasters to see inside weather events to detect motion and dynamics that were invisible to pre-NEXRAD radars. ASOS, which consists of nearly 900 separate ground-based sensor sets, is to provide a portfolio of weather readings, such as temperature and visibility, from more locations and with greater frequency than has been done by human observers.

AWIPS at a Glance

AWIPS is to function as the “central nervous system” of a modernized NWS. That is, it is to be the information processing and display system that forecasters use to integrate, analyze, and graphically view the immense number of weather observations and products that form the basis for each day’s weather and river forecasts and warnings. It is also to be the national communications infrastructure for NWS’ many forecasting offices and centers, connecting them not only to each other but also linking them to the many users of forecasts and warnings throughout the nation.

Using AWIPS’ advanced processing, display, and communications capabilities, NWS expects to fully capitalize on its new observing systems for the first time. Without AWIPS, these observing systems cannot be maximized because the current Automation of Field Operations and Services (AFOS) computer and communications system and the associated weather office display systems cannot accommodate the mountainous data streams that these observing systems now provide. For example, the current system can only accept two satellite images an hour. With AWIPS, GOES-Next images are to be received in real time, meaning that up to eight images can be received and displayed each hour.

Overview of AWIPS’ Expected Impact on NWS’ Mission Performance

According to NWS, AWIPS is to contribute to improvements in the accuracy and timeliness of forecasts and warnings as well as streamlining its operations and downsizing its organization. All told, NWS expects that the combined pieces of the modernization will result in the number of NWS field offices dropping from 254 to 119 and the number of NWS staff falling from 5,100 to 4,678. AWIPS is expected to improve forecaster productivity by allowing forecasters to view disparate data sets in an integrated fashion, perform an assortment of scientific computations on these data sets, and graphically display and interact with these data sets. Currently, such activities are generally performed manually as data from the various observing systems are displayed on multiple screens and assimilated by the forecaster. For example, today, when forecasters want to combine radar and satellite images to view weather pattern movements, the images must be manually overlayed on transparencies. With AWIPS, such integration is expected to occur automatically on the AWIPS workstation with a few simple mouse clicks or keystroke commands. In short, AWIPS is to result in forecasters spending less time physically and mentally manipulating data and more time practicing meteorology.

AWIPS History and Status in Brief

The National Oceanic and Atmospheric Administration (NOAA), which is NWS’ parent agency, began AWIPS in the mid-1980s to replace AFOS. Since then, NWS has invested considerable time and effort in analyzing and defining AWIPS’ requirements, effectively involving users, as we reported in 1993, through hands-on experience with prototypes. In 1992, NOAA awarded the AWIPS development contract to the Planning Research Corporation (PRC). Because of the contractor’s failure to deliver an acceptable AWIPS design, NWS renegotiated the contract in 1995, basically assuming responsibility for development of all hydrology and meteorology application software and assigning the contractor responsibility for delivering the hardware and systems software and for integrating the entire system. To fulfill its responsibility, NWS established joint NWS and contractor applications software (for example, the software that executes atmospheric and hydrological numerical and statistical models, manipulates satellite and radar graphics, etc.) development teams.

NWS’ current plans call for building and integrating AWIPS in seven increments. Thus far, NWS and the contractor have installed a very limited version of AWIPS, the first increment, at three sites to gain some experience in developing, testing, implementing, and operating a limited capability AWIPS. This limited version is also intended to validate selected AWIPS architectural features, such as satellite broadcasts and the central communications and system monitoring “hub.” NWS has begun development of the second increment. NWS estimates that AWIPS will cost $525 million to fully develop and deploy. Deployment of less than the full AWIPS capability to NWS field offices and national centers is now scheduled to begin in 1996. Full AWIPS deployment is scheduled to begin in 1999.

A Hierarchical Description of AWIPS Functions, Capabilities, and Requirements

AWIPS consists of about 22,000 requirements that have been grouped into about 450 higher-level capabilities. These capabilities are described in the AWIPS System/Segment Specification, commonly referred to as the “A-Spec,” which relates about three-fourths of the capabilities to five broad functional areas. The five functional areas are (1) communications, (2) monitoring and control, (3) processing, (4) display and interaction, and (5) data management. The first two functional areas constitute what NWS calls the AWIPS network segment (that is, the national communications infrastructure). The latter three are referred to by NWS as the AWIPS site segment (that is, the functionality applicable to AWIPS sites). Appendix II provides examples of AWIPS capabilities for each functional area. The remaining one-fourth relate to such capabilities as AWIPS’ performance, security, availability, and flexibility. Figure 1 depicts this AWIPS hierarchy.

NWS Cannot Show That All AWIPS’ Proposed Capabilities Are Related to Mission Improvements

Investments in information technology, like AWIPS, should be justified on the basis of whether or not all planned system capabilities will make a clear difference in advancing mission efficiency and effectiveness (for example, improved forecasts and warnings). NWS has not done this for AWIPS. According to NWS officials, they have not explicitly linked either AWIPS requirements or higher-level capabilities to mission improvements, and they have no plans to do so because they claim that other requirements reviews, analyses, and validation activities already provide implicit justification for all AWIPS’ proposed capabilities. We disagree. We carefully reviewed these other activities and while we found them to be valuable for different reasons, they were neither intended to nor do they demonstrate that AWIPS’ full array of capabilities will improve NWS mission effectiveness. As a result, NWS risks unnecessarily spending money on AWIPS capabilities that do not satisfy any of its mission improvement goals—better forecasts, fewer field offices, and fewer staff.

System Capabilities Should Be Validated to Mission Improvements

Office of Management and Budget (OMB) Circular A-130, Management of Federal Information Resources, requires agencies to create and maintain management and technical frameworks that define linkages between mission needs and information technology capabilities. OMB Circular A-109, “Major System Acquisitions,” expands on A-130, requiring federal agencies to make system design decisions based on a review of proposed system functional and performance capabilities contributions to mission needs and program objectives. In effect, agencies developing computer systems, like AWIPS, are to show that proposed system capabilities will produce some mission effectiveness or efficiency gain, like more reliable and timely forecasts or office and staffing reductions.

These requirements are consistent with our recent findings on how leading public and private organizations tie technology investments to measurable mission improvements. We found that successful organizations’ information system investment decisions are tied to explicit and quantifiable mission improvements. By doing so, these organizations know that investing in system requirements will make a difference in mission outcomes.

Ensuring that proposed system capabilities are justified before expensive software development begins requires validating (that is, proving) that system requirements are anchored in user needs, which in turn are grounded in positive mission impacts. To do less increases the chances of spending money on capabilities that, even though desired by users, will not advance the organization’s effectiveness or efficiency. Accordingly, software development guidance advocates assuring traceability from derived system requirements, designs, and implementations to both original user needs and mission needs. Approaches to validating proposed capabilities include performance modeling and prototyping.

Despite Past and Ongoing Requirements Reviews, NWS Has Yet to Justify AWIPS’ Capabilities on the Basis of Mission Results

According to NWS, planned AWIPS capabilities are necessary and will contribute to NWS’ goals of weather forecast and warning improvements, field office consolidation, and staff reductions. However, NWS officials were unable to produce any analysis or associated documentation to validate this claim. Instead, they presented the results of past AWIPS requirements analysis and definition activities and discussed ongoing requirements validation activities that, while useful in their own right, do not justify AWIPS capabilities on the basis of mission improvements. Each of these requirements review activities are discussed below.

The first AWIPS requirements review began in 1984, when NWS initiated the Denver AWIPS Risk Reduction and Requirements Evaluation (DARE) program, an extensive AWIPS prototyping effort to analyze and refine meteorology requirements. Later, NWS augmented the DARE prototype addressed roughly half of AWIPS’ 22,000 requirements. The half that were addressed equates to over three-fourths of AWIPS’ total lines of code. Moreover, even those AWIPS capabilities that were part of the prototypes, with one limited exception, were not explicitly linked to measurable improvements in NWS’ mission effectiveness during these prototyping activities. The one exception is an NWS analysis linking DARE activities occurred at the Norman, Oklahoma, weather office to emulate the operations of a future, modernized weather office. According to NWS, the emulation examined such things as the AWIPS user interface for displaying NEXRAD products, AWIPS’ integration of national weather products (e.g., satellite imagery) with local data (e.g., NEXRAD products), and future weather office staffing levels. However, we reviewed the results of these emulation activities and found no evidence validating AWIPS-specific capabilities on the basis of stated NWS mission goals of better forecasts, fewer weather offices, or less staff. For example, one report concluded that the AWIPS prototype provided capabilities for viewing NEXRAD data that forecasters found “useful,” but does not show the mission outcome of having the capabilities. Another report that was AWIPS-specific concluded that a proposed AWIPS capability known as Interactive Computer Worded Forecast (ICWF), which is later to be replaced by the AWIPS Forecast Preparation System (AFPS), actually decreased rather than increased forecaster productivity and should not be deployed in its current form.

In 1994, in response to the contractor’s earlier mentioned failure to produce an AWIPS design, NWS undertook what it refers to as a “functional decomposition” of the 22,000 AWIPS requirements. In effect, NWS placed these requirements into about 450 capability categories. These categories are the foundation of the AWIPS “A-Spec,” which, as mentioned earlier, is the high-level system specification that further combines most of the 450 capabilities into five broad functional areas. (See figure 1.) Clearly, the development of the “A-Spec” was a valuable undertaking in that it translated the 22,000 AWIPS requirements into a smaller, simpler, more understandable set of high-level functions. However, this translation did not, nor was it intended to, link requirements or capabilities to mission improvements.

Beginning in 1994 and continuing through today, NWS also has been reviewing all AWIPS requirements to identify any that are, in its words, “archaic,” meaning that they are technologically obsolete, duplicative of other AWIPS requirements, or in need of modification to comply with AWIPS’ system design. Thus far, NWS has reviewed about 900 of the 1,000 requirements it dubbed as potentially extraneous and chose to eliminate about 600. The remaining 100 still need to be evaluated. Again, this requirements “scrub” has been and continues to be worthwhile. However, NWS’ above-cited criteria do not address whether the proposed requirement or capability will produce a measurable mission improvement.

At the same time NWS is completing its review of the aforementioned “archaic” requirements, the joint NWS/contractor teams established to develop AWIPS’ application software are examining requirements one last time before building each software module. According to the AWIPS Software Development Plan, the required AWIPS scientific applications in many instances were written several years ago and may still contain ones that are obsolete. However, the NWS official responsible for overseeing the joint development teams stated that the teams’ process for reviewing the requirements does not attempt to validate requirements back to mission improvements. Also, the official stated that the focus of the reviews is on reaching a common understanding on how best to proceed in developing the software module.

Conclusions

NWS clearly needs a new, modern system to support its current operations and allow it to take advantage of the vast data streams now available through its new observing systems. However, whether all of the 450 capabilities it plans for AWIPS are necessary to accomplish this is unknown because the process it has followed in developing AWIPS, while providing for traceability between proposed system capabilities and user expressed needs, does not include validating that these capabilities explicitly and measurably advance NWS’ mission efficiency and effectiveness, which NWS has defined in terms of improved forecasts, fewer field offices, and reduced staffing levels. Validating system capabilities to mission outcomes is vital because it confirms prior to costly software development that proposed system capabilities are grounded in and will contribute to NWS’ mission goals. Unless NWS expands ongoing AWIPS requirements review activities to include demonstrating, either quantitatively or qualitatively, that proposed capabilities measurably advance NWS’ mission abilities, it risks spending money to develop capabilities that are not justified.

Recommendations

We recommend that the Secretary of Commerce direct the NOAA Assistant Administrator for Weather Services to (1) expand ongoing AWIPS requirements review activities to include validation that proposed capabilities are justified on the basis of mission impact and (2) not implement any of those capabilities that are not validated. At a minimum, such validation should include analyses of data and factual accounts from past and ongoing AWIPS prototype experiences that link those proposed capabilities to stated mission improvement goals.

Agency Comments and Our Evaluation

On January 24, 1996, we discussed a draft of this report with NOAA and NWS officials, including the NOAA Associate Administrator for Weather Services, the Deputy Assistant Administrator for Operations, and the Deputy Assistant Administrator for Modernization. In general, these officials did not agree with the report’s conclusions and recommendations, reiterating the NWS positions that were in our draft report. In particular, they stated that extensive AWIPS requirements validation activities have occurred and are ongoing. They also stated that only AWIPS capabilities that are essential to NWS’ mission are being pursued, and that their inability to prove that mission-based requirements validation activities were performed is not sufficient to conclude that AWIPS is not needed. They promised to provide additional documentation to show that AWIPS proposed capabilities are grounded in mission impacts.

We agree that extensive validation activities have occurred and are still ongoing, and we give NWS credit for these activities in the report. Unfortunately, NWS’ validation activities have only dealt with a part of the validation equation and have not validated AWIPS capabilities to its stated mission outcomes of better forecasts, fewer field offices, and fewer staff. This is completely at odds with our recent findings on how leading public and private sector organizations base successful technology investments on whether they produce meaningful improvements in the cost, quality, and timeliness of product and service delivery.

Further, we neither state nor imply that AWIPS is not needed. Rather, we are saying that NWS is spending hundreds of millions of dollars without knowing whether all AWIPS capabilities will contribute to its stated reasons for investing in the system (improving forecasts and reducing field offices and staffing levels). Restated, while we do not question the need to replace AFOS, we do question whether AWIPS, with all the capabilities that NWS currently envisions it providing, should be that replacement. Unless NWS addresses this question, it risks spending money for capabilities that do not advance its mission performance. Fortunately, NWS has the opportunity to perform this validation activity as part of already ongoing and planned requirements reviews. We strongly encourage NWS to take advantage of this opportunity.

We reviewed the additional documentation that NWS officials provided and have included it in the report as further evidence of NWS’ thorough validation of AWIPS capabilities to user needs. However, this documentation does not show that AWIPS capabilities are anchored in mission improvements. We have incorporated other comments made by the officials in the report where appropriate.

We are sending copies of this report to the Secretary of Commerce, the Director of the Office of Management and Budget, and interested congressional committees. Copies will also be made available to others upon request.

Please call me at (202) 512-6240 if you or your staff have any questions concerning this report. Other major contributors are listed in appendix III.

Objective, Scope, and Methodology

The objective of our review was to determine whether NWS’ process for developing AWIPS has demonstrated that all proposed system capabilities will contribute to promised modernization outcomes—improved forecasts, fewer weather offices, or reducing staffing levels. To determine this, we interviewed program officials and reviewed system development documentation to document past and ongoing steps to validate AWIPS requirements. In particular, we reviewed analyses of AWIPS’ prototyping efforts, memoranda from the 1992 rebaselining of the AWIPS requirements, the System/Segment Specification for the National Weather Service Advanced Weather Interactive Processing System, and the Requirements Traceability Document for the AWIPS Hydrometeorological Computer Software Configuration Item. We also sought program officials explanations of how AWIPS requirements are tied to and will result in improved forecasts, weather office closings, and staff reductions.

Concerning ongoing AWIPS requirements reviews, we interviewed NWS staff currently reviewing AWIPS requirements to determine the purpose of the reviews and the criteria being used to assess the requirements. In addition, we interviewed the NWS official overseeing the NWS/contractor teams developing the AWIPS software modules to learn what validation steps and criteria the teams are employing.

We provided a draft of this report to the Department of Commerce for comment. On January 24, 1996, we obtained oral comments from NOAA and NWS officials. These comments have been incorporated in the report as appropriate.

We performed our work at the AWIPS program office, and at NOAA and NWS headquarters offices in Silver Spring, Maryland, from August 1995 through January 1996 in accordance with generally accepted government auditing standards.

Examples of AWIPS Capabilities by Functional Area

This appendix provides examples of AWIPS capabilities for each of the five functional areas. — Acquire data automatically when permitted by the observation systems. — Acquire data through external interfaces at the National Meteorological Center segment, including, polar orbiter data, lightning data, and NEXRAD base and derived products. — Acquire data from network segment external interfaces, including formatted GOES-Next products, NEXRAD summary and winds data, surface and upper air observations, and data from GOES-Next data collection platforms. — Distribute data among AWIPS sites. — Request data (excluding NEXRAD products and satellite imagery) from another AWIPS site. — Specify the distribution control parameters, including data destinations, data to be distributed, data prioritization, and destinations required to acknowledge receipt of a high-priority product. — Disseminate data when requested by an external user. — Disseminate hazardous weather products designated by the user to external users automatically. (continued) — Provide NWS with dedicated, electronic access to the network control facility. — Monitor the integrity and timeliness of data and products that are acquired or disseminated over a central interface. — Notify users when degradations and malfunctions of site equipment and communications interfaces are determined. — Provide an orderly shutdown upon detection of a system failure. — Control the display of information on workstations at other AWIPS sites. — Provide the capability for users to remotely install software at another site. — Provide an interactive, graphical method to allow the user to define two unique alert areas for monitoring NEXRAD data. (continued) — Spatially transform a point and grid of points from one map projection, coordinate system, and grid definition to another by interpolation. — Execute one-dimensional numerical cloud models for forecasting cloud top heights, vertical velocity, and hail sizes. — Execute a numerical model to forecast icing potential for aircraft. — Execute a simplified dam break channel flow model. — Compute tide and water level heights and departures. — Compute extraterrestrial radiation parameters. — Generate combined reflectivity/velocity products from NEXRAD data. — Produce three-dimensional image perspective displays. — Perform image inversions, on a pixel-by-pixel basis. — Perform image-sharpening and edge-enhancement on images. — Display the hydrometeorological field on cross-section and time-section plots using contours, plotted values, and wind symbols. — Produce graphical pilot weather briefing displays, including the depiction of the current and forecast conditions along the flight route plotted on a cross-section context background. — Simultaneously display up to at least eight data windows on each workstation monitor. — Toggle between the components of a combined image. — Zoom in and out on displayed image and graphics products with zoom ratios up to 8:1. — Generate color slides, prints, and transparencies of displayed data. — Step frame-by-frame through an animation loop. — Edit elements of a displayed graphic and its attributes on a workstation image/graphics monitor. (continued) — Retrieve data stored locally. — Specify retrieval criteria, such as all temperatures above a certain threshold value. — Store and retrieve hydrometeorological (for example, satellite data, observational data), cartographic (for example, geopolitical boundaries, topography), site management (for example, region information, maintenance activities), and event data (for example, systems errors, performance parameters). — Archive data at the network segment and all site segments.

Major Contributors to This Report

Accounting and Information Management Division, Washington, D.C.

Rona B. Stillman, Chief Scientist for Computers and Communications Randolph C. Hite, Assistant Director Keith A. Rhodes, Technical Assistant Director David A. Powner, Evaluator-in-Charge Robert C. Reining, Information Systems Analyst Colleen M. Phillips, Information Systems Analyst The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.