Department of Commerce: National Weather Service Modernization and NOAA Fleet Issues
Released: 1999-02-24T00:00:00

Letter

National Weather Service Modernization

In the 1980s NWS began a nationwide modernization program to upgrade observing systems such as satellites and radars, and design and develop advanced computer workstations for forecasters. The goals of the modernization are to achieve more uniform weather services across the nation, improve forecasting, provide better detection and prediction of severe weather and flooding, permit more cost-effective operations through staff and office reductions, and achieve higher productivity. For example, NWS plans to reorganize its field office structure from 256 offices (52 Weather Service Forecast Offices and 204 Weather Service Offices), to 121. As of February 1999, NWS officials told us that 132 offices have been closed.

NWS’ system modernization includes four major systems development programs, which are expected to collectively cost about $4.5 billion. I would like to briefly describe each.

Next Generation Weather Radar (NEXRAD). This is a program to acquire 166 Doppler radars. Largely deployed, these radars have helped NWS increase the accuracy and timeliness of warnings for severe thunderstorms, tornadoes, and other hazardous weather events. The reported cost of this program is just under $1.5 billion.

Next Generation Geostationary Operational Environmental Satellite (GOES-Next). This is a program to acquire, launch, and control five geostationary satellites, GOES-I through GOES-M, which assist in the mission of identifying and tracking severe weather events, such as hurricanes. The first satellite in the current series was launched in 1994 and the fifth is scheduled for launch in 2002. The total cost for these five satellites, including launch services and ground systems, is estimated to be just under $2 billion.

Automated Surface Observing System (ASOS). This is a program to automate and enhance methods for collecting, processing, displaying, and transmitting surface weather conditions, such as temperature and precipitation. The system is planned for installation at 314 NWS locations. Estimated costs for the ASOS Program are about $350 million, which includes the NWS units and another 679 units for the Federal Aviation Administration and the Department of Defense.

Advanced Weather Interactive Processing System (AWIPS). This program integrates, for the first time, satellite, radar, and other data to support weather forecaster decision-making and communications; it is the linchpin of the NWS modernization. AWIPS, which was originally scheduled to be developed incrementally in a series of six modules, or builds, is currently set to be deployed to 152 locations after the fourth build by the end of June 1999.

In 1995 we designated the NWS modernization a high-risk area for the federal government because of its estimated $4.5 billion cost, its complexity, its criticality to NWS’ mission of helping to protect life and property through early forecasting and warnings of potentially dangerous weather, and its past problems--documented in several of our reports. Our 1997 high-risk series reported that although the development and deployment of the observing systems associated with the modernization were nearing completion, unresolved issues remained. These concerned the systems' operational effectiveness and efficient maintenance. For example, new radars were not always up and running when severe weather threatened, and ground-based sensors fell short of user expectations, particularly during active weather. We recommended that NWS correct shortfalls in radar performance, and define and prioritize all ground-based sensor corrections according to user needs.

Some of our radar and ground-based sensor performance concerns were addressed, while others remain. We recently reported that a NEXRAD unit in southern California failed to consistently meet NWS’ own NEXRAD availability requirement, and recommended that the Weather Service correct the problem such that the radar meets availability requirements. NWS agreed, and has several activities planned to bring about such improvement.

While there have been specific performance problems, NWS reports that the new radars and satellites overall have enabled it to generate better data and greatly improved forecasts and warnings. We continue to view the NWS modernization as a high-risk area, however, for two primary reasons: (1) NWS lacks an overall architecture to guide systems development and (2) the final piece of the modernization--AWIPS (the forecaster workstations that will integrate weather data from NEXRAD, GOES-Next, and ASOS)--has not yet been deployed. At this point I would like to discuss these issues in more detail.

NWS Modernization Lacks Overall Systems Architecture

A systems architecture is an essential tool for guiding effective and efficient systems development and evolution. We initially reported in 1994 that the NWS modernization needed such an overall technical blueprint; NWS agrees--and is currently working on one. Until such an architecture is developed and enforced, the modernization will continue to be subject to higher costs and reduced performance. This is an important point as component systems continue to evolve to meet additional demands and take advantage of improved technology. The Assistant Administrator for Weather Services shares this view, and said recently that NWS plans to intensify its efforts to develop a systems architecture.

Delivery of AWIPS Remains a Concern

Until AWIPS is fully deployed and functioning properly, NWS will not be able to take full advantage of the $4.5 billion total investment it has made in the modernization. Over the past several years, we have reported that AWIPS has encountered delays and cost increases due to design problems and management shortcomings and have made several recommendations to improve management of this critical component of the modernization. NWS has acted on most of our recommendations. I would like to now update you on AWIPS’ cost, schedule, software development, and maintenance.

The cost to develop AWIPS was estimated at $350 million in 1985; a decade later, that figure had risen to $525 million. However, in testimony and a report issued in 1996, we pointed out the inaccuracy of this $525 million estimate due to the omission of several cost factors, including known contract increases. The Department of Commerce later committed to a $550 million funding cap. Yet as we testified in April 1997, it would prove extremely difficult for NOAA to develop and deploy AWIPS within the $550 million cap if any problems were encountered. Given the size and complexity of the development--and recognizing that even managed risks can turn into real problems--we testified that such problems were likely to occur and that costs would likely exceed $550 million.

In accordance with a recommendation we made in 1996, the department contracted for an independent cost estimate of AWIPS because of the uncertainty about whether it could be delivered within the $550 million cap given the increased software development expenses. According to the assessment dated February 2, 1998, the likely cost to complete AWIPS through its final build--build 6--was $618 million.

In March 1998, we reported that although AWIPS was planned for full deployment through build 6 in 1999--at 152 locations nationwide--that schedule is now in doubt. The latest schedule calls only for build 4-- actually build 4.2--to be completed in June, within the $550 million cap. Also as we testified last year, completion dates for builds 5 and 6 were uncertain because NWS wanted to ensure that requirements for those modules were not extraneous to mission needs, in order to minimize future cost increases. This reflects a recommendation we made in 1996 for all AWIPS builds. In August 1998, an independent review team reported that build 5 requirements are essential to NWS’ core mission and that the cost to complete should range from an additional $20 to $25 million above the $550 million cap. The team concluded that build 6 requirements should not be pursued, however, because they “resemble capabilities desired, rather than requirements.”

According to the AWIPS program manager, deployment of build 4.2 will result in improved forecasts and warnings, a reduction of 106 staff, and the decommissioning of the current Automation of Field Operations and Services (AFOS) system. The program manager added that build 5 will be pursued in order to realize expected further improvements in weather forecasts and warnings, a reduction of an additional 69 staff, and the decommissioning of the NEXRAD workstations. Schedules for build 5 have not yet been developed. To help ensure that build 4.2 will be delivered within the cap, the Assistant Administrator for Weather Services has contracted with an independent accounting firm to verify program expenditures.

The most critical risk factors underlying questions about AWIPS' future relate to software development. We have frequently reported on this and made several recommendations to improve AWIPS’ software development processes. Software quality is governed largely by the quality of the processes used to develop it; however, NWS’ efforts to develop AWIPS software have lacked defined development processes. Such processes are all the more essential because of NWS’ increased use of software code developed internally at NOAA’s Forecast Systems Laboratory (FSL) in Boulder, Colorado--a research and development facility that primarily develops prototype systems. This software code has not been developed according to the rigorous processes commonly used to develop production- quality code. Failure to adhere to these processes may result in unstable software that will continue to cause cost increases and schedule delays.

The cost assessment delivered in February 1998 also found risk inherent in the development of builds 4 through 6 because of the transitioning of FSL- developed software to AWIPS and the uncertainty surrounding requirements for these builds. NWS officials have acknowledged these software development process weaknesses, and have told us that they continue to strengthen these processes. For example, NWS reports that all AWIPS software, both that developed by the government and the contractor, is being controlled under a common configuration management process.

Another risk area concerns the network control facility, which provides the ability to monitor and maintain AWIPS sites across the country from a single location. As we testified last year, through build 3, AWIPS was still experiencing difficulty with the central location's ability to detect and respond to problems. We further testified that since these problems concerned only a limited number of sites that as more sites come on line, problems can be expected to increase. NWS officials have acknowledged that the poor performance of the network control facility continues to be a prime concern, have sought the advice of external consultants, and have initiated a number of actions to improve performance of this facility.

Progress Made on Year 2000, But Critical Testing and Contingency Planning Efforts Remain

Finally, a critical risk area is whether the AWIPS builds--and, indeed, all modernization components--will be Year 2000 compliant. AWIPS to date is not Year 2000 compliant. Build 4.2--set for completion this June--is intended to make all AWIPS applications Year 2000 compliant. In the event it is late, NWS has renovated its current system, Automation of Field Operations and Services, to be ready as a potential backup.

Yet even if Year 2000 compliance ceases to be an issue with build 4.2, NWS’ companion modernization systems will need to be compliant because of the amount of data they exchange. NWS reports that five of the six mission-critical systems that interface with AWIPS are already Year 2000 compliant, on the basis of individual systems tests. The remaining system is scheduled to be compliant by March 31, 1999, according to the Department of Commerce’s February 1999 Quarterly Year 2000 Progress Report to the Office of Management and Budget.

To ensure that these mission-critical systems can reliably exchange data with other systems and that they are protected from errors that can be introduced by external systems, NWS has begun to perform end-to-end testing. These tests include multiple Weather Service systems working together and critical interfaces with the Department of Defense and the Federal Aviation Administration. NWS plans to continue to conduct this end-to-end testing through March of this year. The final report on the results of these tests is scheduled to be issued this May.

We suggest that NWS consider conducting additional end-to-end testing after the final version of AWIPS is delivered, which is currently scheduled for this June. Currently, NWS is using a prior version of AWIPS in its end-to- end testing—a version that continues to be modified as AWIPS’ system- level testing progresses. Testing with the final version of AWIPS will help to ensure that the production system that will be running in the year 2000 will work with its interrelated systems.

To reduce the risk and potential impact of Year 2000-induced information systems failures on the Weather Service’s core business processes, it is critical that NWS have contingency plans in place that will help ensure continuity of operations through the turn of the century. Without such plans, NWS will not have well-defined processes to follow in the event of failures. NWS depends on data provided by other federal agencies as well as on services provided by the public infrastructure (e.g., power, water, voice and data telecommunications). One weak link anywhere in this chain of critical dependencies could cause major disruption to NWS operations. Given these interdependencies, it is imperative that contingency plans be developed for all critical core business processes. According to NWS’ Year 2000 program manager, the Weather Service has begun drafting contingency plans for three core business processes: those that (1) observe weather data, (2) produce forecasts and warnings, and (3) disseminate data. It is essential that NWS develop these business continuity and contingency plans expeditiously, and test these plans to ensure that they are capable of providing the level of support needed to allow continued functioning of NWS’ core business processes in the event of failure.

As noted in our business continuity and contingency guide, another key element of such a plan is the development of a zero day or day one risk reduction strategy and, more generally, procedures for the period between December 1999 and early January 2000. Key aspects of this strategy can include the implementation of (1) an integrated control center, whose purposes include the internal dissemination of critical data and problem management and (2) a timeline that details the hours in which certain events will occur (such as when backup generators will be started) during the late December and early January rollover period. To date, NWS has no such strategy. We suggest that the development of such a risk reduction strategy be undertaken.

In conclusion, NWS has made progress on the development and operational testing of the forecaster workstations and its Year 2000 testing and contingency planning. However, cost, schedule, and technical risks associated with the workstations continue to be concerns. Further, the results of NWS’ Year 2000 end-to-end testing and business continuity and contingency plans are expected to be delivered soon.

Pursuing the Most Cost-Effective Alternatives for Acquiring NOAA’s Marine Data

NOAA has an aging in-house fleet of 15 ships that are used to support its programs in fisheries research, oceanographic research, and hydrographic charting and mapping. Most of NOAA’s ships are past their 30-year life expectancies and many of them are costly and inefficient to operate and maintain and lack the latest state-of-the-art technology. NOAA’s ships are managed and operated by a NOAA Corps of about 240 uniformed service commissioned officers who, like the Public Health Service Corps, perform civilian rather than military functions but are covered by a military-like pay and benefits system.

For more than a decade, congressional committees, public and private sector advisory groups, the National Performance Review, the Commerce Office of Inspector General (OIG), and our office have urged NOAA to aggressively pursue cost-effective alternatives to its in-house fleet of ships. We have also reported and testified on issues relating to NOAA’s Commissioned Corps that manages and operates the in-house fleet of ships.

We reported on NOAA’s fleet operations and fleet modernization needs in 1986 and again in 1994 and summarized our earlier work, the Commerce OIG’s work, and the Department of Commerce’s corrective actions in summary reports in January 1998 and January 1999. As part of our recent special performance and accountability series of reports, we identified the NOAA fleet as one of four major performance and management issues confronting the Department of Commerce. As early as 1986 we reported that NOAA needed to develop more definitive information on private ships’ availability, capability, and costs before taking any action to deactivate NOAA’s ships. In 1994, we reported that NOAA (1) lacked the financial and operational data it needed to adequately assess whether chartered and contracted ships could cost effectively meet the needs of its programs and (2) had no assurance that its fleet modernization plan represented the most cost effective means of meeting future program requirements. Consequently, we recommended that NOAA take several actions to ensure that all viable and cost-effective options for accomplishing its program missions are considered in making decisions on future fleet modernization.

The Commerce OIG has also reported and testified several times on the NOAA fleet modernization issue, identified the fleet as one of the top 10 management problems facing the Department of Commerce in April 1997, January 1998, and again in December 1998, and continues to believe that NOAA could and should be doing more to pursue cost-effective alternatives to its in-house fleet of ships for acquiring marine data. Following reports by us, the Commerce OIG, and others, the Department of Commerce initially identified the NOAA fleet as a material weakness in its annual Federal Managers’ Financial Integrity Act (FMFIA) report for fiscal year 1990. It remains a material weakness today.

Since 1990, NOAA has developed several fleet replacement and modernization plans that call for investments of hundreds of millions of dollars to upgrade or replace these ships, and each has been criticized by the Commerce OIG for not pursuing alternative approaches strongly enough. For example, in a 1996 program evaluation report on NOAA’s $1 billion 1995 fleet modernization plan, the OIG recommended that NOAA terminate its fleet modernization efforts; cease investing in its ships; immediately begin to decommission, sell, or transfer them; and contract for the required data or ship services.

In response to these criticisms, NOAA now says that it has taken steps to improve the cost efficiency of its fleet and significantly increased its outsourcing for these services from about 15 percent in 1990 to over 40 percent today. According to NOAA, for example, it has removed seven ships from service and brought one new and two converted Navy ships into service since 1990, now outsources for about 46 percent of its research and survey needs, and expects to further increase its use of outsourcing to about 50 percent over the next 10 years.

Although NOAA apparently has made progress in reducing the costs of its fleet and outsourcing for more of its research and data needs, NOAA continues to rely heavily on its in-house fleet and still plans to replace or upgrade some of these ships. In this regard, the President’s budget for fiscal year 2000 proposes $52 million for construction of a new fisheries research ship and indicates that NOAA plans to spend a total of $185 million for four new replacement ships over the 5-year period ending in fiscal year 2004--$52 million in 2000, $51 million in 2001, $40 million in 2002, $40 million in 2003, and $2 million in 2004. We have not had an opportunity to review the latest studies of NOAA’s fleet modernization efforts or NOAA’s acquisition plan for its fisheries research mission. Thus, we do not know whether or not NOAA’s proposed replacement ships are the most cost- effective alternative currently available for meeting these fisheries research needs.

In addition to its proposed acquisitions, NOAA also continues to repair and upgrade its aging fleet of existing ships. Since 1990, it has repaired and upgraded seven of its existing ships and plans to repair and upgrade two more in 1999. According to the President’s recent budget requests, NOAA spent $12 million in 1996 and $13 million in 1997 to modernize, convert, and replace its existing ships. Also, it spent $21 million on fleet maintenance and planning in 1998 and expects to spend $13 million in 1999 and $9 million in 2000.

The question of the viability of the NOAA fleet is entwined with the issue of the NOAA Corps, which operates the fleet. In 1995, the National Performance Review, noting that the NOAA Corps was the smallest uniformed service and that the fleet it commanded was obsolete, recommended that the NOAA Corps be gradually reduced in numbers and eventually eliminated. We reported in October 1996 that the NOAA Corps generally does not meet the criteria and principles cited by the Department of Defense for a military compensation system. We also noted that other agencies, such as the Navy, the Environmental Protection Agency (EPA), and the Federal Emergency Management Agency (FEMA), use federal civilian employees or contractors to carry out duties similar to the functions that NOAA assigns to the Corps. Commerce developed a plan and legislative proposal to “disestablish” or civilianize the NOAA Corps in 1997, but the Congress did not adopt this proposal.

According to NOAA and to the Department of Commerce’s annual performance plans for fiscal years 1999 and 2000 under the Results Act, the NOAA Corps has been downsized from over 400 officers in fiscal year 1994 to about 240 at the beginning of fiscal year 1999, achieving gross annual cost savings of at least $6 million. In June 1998, NOAA announced a new restructuring plan for the NOAA Corps. NOAA’s plan focused on the need for a NOAA Commissioned Corps of about 240 officers. NOAA’s June 1998 restructuring plan also called for a new civilian director of the NOAA Corps and a new recruiting program.

However, the Congress had other ideas. The Omnibus Appropriations Act for fiscal year 1999 set the number of NOAA Corps officers at 250. Subsequently, the Governing International Fishery Agreement Act (Public Law 105-384, approved November 13, 1998) made other changes in NOAA’s proposed restructuring plan. This act authorized a NOAA Corps of at least 264 but not more than 299 commissioned officers for fiscal years 1999 through 2003, requires that a uniformed flag officer be the NOAA Corps’ operational chief, and directed the Secretary of Commerce to lift the then- existing recruiting freeze on NOAA Corps officers. According to the NOAA Corps, it expects to have about 250 commissioned officers by the end of fiscal year 1999.

In summary, NWS faces significant challenges this year—both in deploying the initial version of AWIPS and in addressing the Year 2000 problem. Longer term, NWS still needs to develop an overall systems architecture and to develop AWIPS’ build 5 requirements since they are essential to NWS’ core mission. In the NOAA fleet area, continuing congressional oversight of NOAA’s budget requests for replacement or upgraded ships is needed to ensure that NOAA is pursuing the most cost-effective alternatives for acquiring marine data.

This concludes our statement. We would be happy to respond to any questions that you or other members of the Subcommittee may have at this time.

The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary, VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.