Weather Forecasting: Unmet Needs and Unknown Costs Warrant Reassessment of Observing System Plans
Published: 1995-04-21T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a congressional request, GAO reviewed the National Weather Service's (NWS) Automated Surface Observing System (ASOS), focusing on: (1) the extent to which NWS has addressed ASOS operational problems; (2) the costs of resolving these operational problems; and (3) whether NWS plans for enhancing ASOS are reasonable.
section_title: What GAO Found
paragraphs: GAO found that: (1) six of the eight sensors in the ASOS system do not meet key performance specifications; (2) ASOS shortfalls are caused by contractor failure to deliver products that meet specifications and government failure to furnish sufficient equipment; (3) ASOS reliability problems surfaced after deployment because NWS did not perform reliability testing prior to deployment; (4) NWS does not have adequate personnel or integrated information systems for it to isolate and correct ASOS failures at Federal Aviation Administration sites; (5) ASOS does not satisfy the weather observation needs of many users; (6) ASOS users state that incorrect ASOS observations could risk aviation efficiency and safety, and skew national climate research; (7) in March 1995, the National Oceanic and Atmospheric Administration (NOAA) resumed the commissioning of observers on the basis of expected system and equipment improvements; (8) although NWS has recently begun to recognize user needs that go beyond ASOS specifications, it has not yet determined how it will enhance or supplement ASOS to address user needs; (9) NWS cannot reliably assess whether system enhancements will continue to be cost beneficial without a complete cost estimate; (10) ASOS problems have delayed NWS plans for releasing human weather observers; and (11) although ASOS officials stated that user needs will be met before observers are released, NWS does not have a strategy to ensure that the information needs of all users will be met.

section_title: Letter
paragraphs: This report responds to your request that we review the National Weather Service’s (NWS) Automated Surface Observing System (ASOS). As agreed with your offices, we focused on ASOS’ operational problems and NWS’ efforts to resolve them, the cost of resolving these problems, and whether NWS’ plans for implementing ASOS make sense in light of these problems.
We are providing copies of this report to the Secretary of Commerce; the Secretary of Transportation; the Director, Office of Management and Budget; and interested congressional committees. Copies will also be made available to others upon request.
Please call me at (202) 512-6253 if you or your staffs have any questions concerning the report. Other contributors to this report are listed in appendix III.
subsections: section_title: Introduction
paragraphs: The National Weather Service (NWS) is a component of the National Oceanic and Atmospheric Administration (NOAA), within the Department of Commerce. NWS’ basic mission is to provide weather and flood warnings, public forecasts, and advisories primarily for the protection of life and property. NWS’ operations also support other agencies’ missions and the nation’s commercial interests. For example, NWS provides specialized forecasts to support aviation safety and the agricultural and marine industries. To carry out its mission, NWS uses a variety of systems and manual processes to collect, process, and disseminate weather data to and among its network of field offices and regional and national centers. Many of these systems and processes are outdated. For example, some radar equipment currently in use dates back to 1957. These radars experience frequent and long-lasting outages, and some replacement parts are no longer available.
During the 1980s, NWS initiated a program to modernize its systems and restructure its field offices. The goals of this modernization program are to achieve more uniform weather services across the nation, improve forecasts, provide more reliable detection and prediction of severe weather and flooding, permit more cost-effective operations, and achieve higher productivity. The modernization program includes four new major system developments: the Next Generation Weather Radar (NEXRAD), the Next Generation Geostationary Operational Environmental Satellite (GOES-Next), the Advanced Weather Interactive Processing System (AWIPS), and the Automated Surface Observing System (ASOS). It also includes upgrades to existing systems and several smaller new system development projects. The total cost of the modernization is estimated to exceed $4.5 billion. We have previously reported on NWS’ modernization.Additionally, we designated NWS’ modernization as a high-risk information technology initiative in our 1995 report series on high-risk programs.
NWS plans to restructure its field offices in association with the modernization program. This restructuring will consolidate the current field structure of 52 Weather Service Forecast Offices, 204 smaller Weather Service Offices, 13 River Forecast Centers, and 3 National Centers into 118 Weather Forecast Offices, 13 River Forecast Offices,  13 Data Collection Offices, and 9 National Centers.
Three other agencies are participating with NWS in the modernization. The Federal Aviation Administration (FAA) and the Department of Defense (DOD) are major users of weather data and have formed tri-agency programs with NWS to manage and fund the development and deployment of both NEXRAD and ASOS. Similarly, the National Aeronautics and Space Administration (NASA) is managing the development and procurement of GOES-Next.
Because of the importance of timely and accurate weather forecasts and warnings to our nation’s commerce and its citizens’ safety, the Congress has been keenly interested in the modernization, particularly in preserving the existing quality of weather service. As a result, the Congress passed Public Law 100-685 in 1988, and Public Law 102-567, Title VII, in 1992. This legislation states that the Secretary of Commerce shall not close, consolidate, automate, or relocate any field office until the Secretary certifies to the Congress that such action will not result in any degradation of weather service to the public.
subsections: section_title: ASOS: A Brief Description
paragraphs: ASOS is a system of sensors, computers, display units, and communications equipment intended to automate the ground-based observation and dissemination of weather information nationwide. This weather information includes data on temperature and dew point, visibility, wind speed and direction, pressure, cloud height and amount, and types and amounts of precipitation. Figure 1.1 depicts the specified configuration of ASOS sensors and describes their functions.
According to NWS, ASOS offers considerable advantages over the current manual surface observation practices, including uniform measurements across all sites and times, continuous observation and reporting, and more observing sites nationwide. ASOS is also intended to eventually replace human observers at weather service offices and at airports. However, ASOS is not designed to detect all weather conditions that human observers have historically provided. Weather conditions that ASOS does not observe include thunderstorms, tornadoes, volcanic ash, hail, virga, snow amount and depth, cloud layers above 12,000 feet, ice pellets, drizzle, blowing snow, blowing dust, and blowing sand. NWS plans to continue to provide this information to its users through a combination of remote sensing systems (e.g., radar, satellite, and lightning detection system), additional ASOS sensors, and manual observation by non weather service observers, such as volunteers, on-call cooperators, and/or FAA personnel.
subsections: section_title: ASOS Status and Plans
paragraphs: NOAA, as the lead agency on the ASOS program, has contracted with AAI Corporation to purchase a minimum of 868 units, with an option to buy an additional 832 units. Through the base contract, NWS, FAA, and DOD plan to procure 245, 537, and 86 ASOS units, respectively. As of December 1994, NWS had purchased 617 units for the three agencies. Of these, 491 have been accepted, and 47 have been commissioned. NWS commissions a system when it provides the official weather observation. However, human observers still augment ASOS observations at those sites that have historically used human observers. In these cases, the human observers continue to deliver some of the information ASOS cannot provide (such as thunderstorms and tornadoes), and provide correct observations should ASOS err or fail. (Table 1.1 provides more information on the status and number of ASOS units for each agency).
NWS plans to purchase and install all 868 units by fiscal year 1997. In June 1993, NOAA estimated that developing and deploying the base ASOS program through fiscal year 1996 for all three agencies would cost about $340 million. NWS now estimates that the base program will cost $351 million through fiscal year 1997 due to FAA changes to the program. The three agencies spent roughly $235 million to develop and deploy ASOS through fiscal year 1994, and plan to spend about $116 million to procure and deploy the remaining ASOS units in the base program through fiscal year 1997.
section_title: ASOS Is Intended to Replace Human Observers
paragraphs: Until an ASOS site is commissioned, human observers are responsible for providing official NWS surface weather observations. These observations are used by meteorologists, hydrologists, climatologists, and the aviation community to provide weather forecasts, research climate changes, and conduct aviation operations. Within the aviation community, these observations are necessary for aircraft to take off, land, or be dispatched to an airport. As part of its modernization and associated restructuring, NWS expects that ASOS will assume most surface observing responsibilities and replace human observers at about 175 weather offices and airports. In addition, according to the FAA manager responsible for defining air traffic requirements for weather sensors, FAA intends for ASOS to replace its contract observers at about 194 sites. To date, neither FAA nor NWS has replaced any of their human weather observers.
section_title: ASOS Management Structure
paragraphs: Three organizations play key roles in the ASOS program. NOAA’s System Acquisition Office is responsible for the ASOS acquisition. NWS’ ASOS program office manages the overall program. Its responsibilities include overseeing system development and deployment, coordinating the program with FAA and the Navy, and investigating system enhancements. Finally, NWS’ Office of Systems Operations (OSO) is responsible for operating and maintaining ASOS systems. Within OSO, ASOS Operations and Monitoring Center (AOMC) is responsible for on-line monitoring of the quality of ASOS observations and dispatching maintenance technicians for operational ASOS units.
section_title: Objectives, Scope, and Methodology
paragraphs: The objectives of our review were to determine (1) what ASOS problems exist and how effectively NWS is resolving them, (2) the cost of resolving these problems, and (3) whether NWS’ plans for implementing ASOS make sense in light of these problems.
To determine what ASOS problems exist, we obtained and analyzed (1) the ASOS program office’s problem report database, (2) summaries of issues compiled by an NWS task team charged with resolving ASOS problems, (3) action plans on problematic sensors developed by this NWS task team, (4) NWS’ system performance reports, (5) relevant FAA Unsatisfactory Condition Reports, (6) relevant NASA Aviation Safety Reporting System reports, (7) ASOS discrepancy reports from two weather service offices and three air traffic control towers that representatives from the National Air Traffic Controllers Association (NATCA) and the National Weather Service Employees Organization (NWSEO) recommended we contact, and (8) ad hoc problem reports and letters from weather observers, air traffic controllers, and air dispatchers; discussed meteorologists’ requirements and system performance with and obtained related documentation from NWS’ Test and Evaluation Branch in Sterling, Virginia; NWS’ Office of Meteorology; 14 NWS weather service offices across the country; and the NWSEO; discussed climatologists’ requirements and system performance with and obtained related documentation from NOAA’s National Climate Data Center, NWS’ Climate Data Continuity Project, the Colorado State University Department of Atmospheric Science, and the ASOS Climate Working Group; and discussed aviation requirements and system performance with and/or obtained related documentation from the National Research Council, FAA’s Air Traffic Plans and Requirements Service, NATCA, the Air Transport Association, the Airline Dispatchers Federation, and 13 individual air traffic controllers from airport towers referred to us by NATCA.
To determine the effectiveness of NWS’ problem resolution efforts, we reviewed policies and procedures and interviewed program officials to document NWS’ processes for identifying and resolving ASOS problems, obtained and assessed documentation on problem resolution efforts and discussed these efforts with NWS and FAA program officials, and assessed a random sample of NWS’ database of 499 problem reports to identify (1) how these reports were tracked and their resolution efforts documented and (2) how satisfied the originators of the problem reports were with NWS’ resolution efforts.
To determine the cost of resolving ASOS problems, we requested cost estimates for resolving known problems. We then reviewed available documentation on the costs to fix these problems and analyzed the current ASOS project cost estimate to determine what costs were and were not included.
Last, to determine whether plans for implementing ASOS make sense in light of these problems, we discussed deployment and commissioning plans and NWS’ analysis of alternative deployment plans with program officials, and we requested copies of any analyses performed. Additionally, we compared NWS’ deployment plans with plans for resolving ASOS problems for inconsistencies.
In performing our work, we used our System Assessment Framework methodology for guidance on assessing operational systems. We also used our research on the best information management and technology practices of leading organizations as published in our report, Executive Guide: Improving Mission Performance Through Strategic Information Management and Technology, (GAO/AIMD-94-115, May 1994).
We performed our work at the Department of Commerce in Washington, D.C.; NOAA and NWS headquarters in Silver Spring, Maryland; NOAA’s Sterling Research and Development Center in Sterling, Virginia; NWS weather offices in Colorado Springs and Denver, Colorado; FAA headquarters in Washington, D.C.; FAA’s air traffic control tower at the Colorado Springs Airport in Colorado Springs, Colorado; Colorado State University in Fort Collins, Colorado; the National Climate Data Center in Asheville, North Carolina; and the Air Transport Association, National Weather Service Employees Organization, and National Air Traffic Controllers Association headquarters in Washington, D.C.
Our work was performed between October 1993 and March 1995, in accordance with generally accepted government auditing standards. We requested written comments from the Department of Commerce and the Department of Transportation, but none were provided within the allotted response period. However, we obtained oral comments from senior NOAA and FAA officials, including the NOAA Comptroller, the NOAA Assistant Administrator for Weather Services, the FAA Weather Sensor Systems Manager, and the FAA Product Lead of the Terminal Products Team. Their comments are presented and addressed in chapter 6.
section_title: ASOS Falls Short of Specified Requirements and Corrective Actions Not Yet Completed
paragraphs: ASOS units are not meeting all specified requirements. NWS recognizes this and has actions under way or planned to correct the shortfalls. Whether or not these actions prove successful, however, remains to be seen. Until then, fielded ASOS units will provide some inaccurate or incomplete weather observations. If not caught and corrected by weather observers, these observations could adversely impact aviation operations, climate research, and the general public. Despite ASOS’ inability to fully meet specified requirements, NWS chose to accept, deploy, and rely on the system because testing prior to the decisions did not disclose the problems. To NWS’ credit, it has historically been effective at eventually resolving ASOS shortfalls in meeting specifications.
subsections: section_title: ASOS Currently Does Not Meet Some Important Specified Requirements, but Steps Are Under Way to Address Shortfalls
paragraphs: Specifications define the minimum system functions and performance levels needed to satisfy user needs. As such, they provide the baseline against which to gauge how successful a system developer is in providing promised capabilities. Failure to satisfy specifications means that mission needs are not fulfilled unless system shortfalls are corrected or work-arounds are implemented. Meeting specified requirements is thus critical to the success of any system.
While ASOS meets many specifications, it currently does not meet some specified requirements on most ASOS sensors, requirements for overall system reliability, and requirements for performance monitoring of FAA ASOS units. In some cases, this is because the contractor failed to provide the promised capability, while in other cases, it is because ASOS components provided to the contractor as government-furnished equipment do not perform as required, thus preventing the contractor from delivering what was promised. ASOS program officials told us that most of these shortfalls did not surface during acceptance testing, and with the exception of the freezing rain sensor discussed below, have only been brought to light as a result of ASOS’ extended operational use during harsh weather.
NWS and the ASOS contractor are taking steps to address most of these current shortfalls. In fact, ASOS units now being produced include sensor upgrades to correct some problems. However, these upgrades do not address all known problems, and earlier deployed units have not been retrofitted with these upgrades. As a result, ASOS systems that are currently operational provide some inaccurate or incomplete weather observations. Unless caught and corrected by weather observers, these weather reports are provided to the aviation community, climatologists, hydrologists, meteorologists, and the general public. This could adversely impact aviation safety, environmental policy decision-making, or even an individual’s preparation for environmental conditions.
Because of ASOS’ problems, the NOAA Assistant Administrator for Weather Services decided in November 1994 to temporarily halt commissioning additional units. NWS officials identified three areas that needed to be addressed in order to resume commissionings. These included (1) installing the improved wind sensors, (2) correcting the precipitation accumulation sensor problems, and (3) improving the availability of ASOS spare parts. Also in November 1994, FAA agreed with its union to halt ASOS commissionings at FAA-sponsored towered airport sites for 7 months, during which time both organizations would evaluate ASOS’ suitability for air traffic operations. This evaluation is ongoing.
In March 1995, NOAA’s Assistant Administrator for Weather Services approved the resumption of commissionings beginning in April 1995. This decision was based on NWS’ satisfaction of two of the three criteria. Specifically, the wind sensor modification was approved for implementation and an adequate supply of spare parts is expected to be delivered in April 1995. The precipitation accumulation sensor has not yet been approved for implementation.
subsections: section_title: Most ASOS Sensors Do Not Meet Specifications
paragraphs: Of ASOS’ eight sensors, four that were developed by the contractor and two that were provided in limited quantities to the contractor as government-furnished equipment do not meet contract specifications. According to program office reports, two other contractor-developed sensors (the visibility and pressure sensors) meet specifications. Thus, promised capabilities have not been delivered on six of eight ASOS sensors.
The contractor-developed sensors that do not meet specifications are the wind, precipitation identification, precipitation accumulation, and freezing rain sensors. Specifically, (1) the wind sensor falls short of specifications for reporting wind speed and direction accurately, and operating during freezing conditions, (2) the precipitation accumulation sensor does not meet accuracy specifications for liquid precipitation amounts, (3) the precipitation identification sensor does not meet specifications for operating during heavy snow and icing conditions, and (4) the freezing rain sensor failed acceptance tests and thus has not yet been accepted by the program office or deployed with the baseline ASOS configuration. Program officials stated that, with the exception of the freezing rain sensor, these problems were not discovered during system acceptance testing.
NWS officials acknowledged that all but one of these sensors do not yet meet specified requirements. However, they stated that the precipitation identification sensor’s problem is not a failure to meet the specification. We disagree. We found that the problem—snow blockage—is identified in NWS’ trouble report system as a contract deficiency, meaning that any sensor modifications to address this problem are the contractor’s responsibility.
The government-provided sensors that do not meet specifications are the temperature and dew point sensor and the ceilometer. The temperature and dew point sensor does not meet reliability requirements, and the ceilometer fails more frequently than the specification permits. The shortfalls of each of these six sensors in meeting specifications and NWS’ resolution plans are discussed in detail in appendix I.
Inaccurate reports, according to a representative for the Airline Dispatchers Federation, could lead a pilot to approach an airport during underreported wind gusts, incurring unexpected safety risks. Similarly, a University of Maryland climatologist stated that inaccurate precipitation reports could lead researchers to incorrectly evaluate flood and drought frequencies, intensities, and durations, and could result in ineffectively designed irrigation systems, dams, and other structures. Finally, a senior NWS meteorologist stated that inaccurate dew point observations could contribute to a forecast for clear weather, when in fact a hazardous thunderstorm is brewing.
NWS has steps under way to address most of these sensor shortfalls, but these actions are not yet complete. The program office, which is responsible for the units being produced, has had the contractor develop modifications to resolve shortfalls with the wind sensor, the precipitation accumulation sensor, and the precipitation identification sensor; has tested the modifications; and has incorporated these modifications in ASOS units now being produced. NWS’ Office of System Operations (OSO), which is responsible for the units once they are commissioned and which did not participate in the program office testing, is testing the modifications to these three sensors in an operational setting before approving them for implementation on commissioned systems. In February 1995, OSO approved the wind sensor modification for implementation on commissioned systems, based on preliminary testing. However, OSO is continuing to test the modification.
Additionally, the program office is currently testing modifications to the freezing rain sensor and ceilometer and evaluating how best to improve the wind direction problem, excessive ceilometer failures, and the dew point sensor reliability problems.
Until NWS completes actions to resolve ASOS shortfalls, their success cannot be ascertained. In the interim, ASOS will not provide its full complement of promised capabilities and will sometimes produce inaccurate or incomplete weather reports.
section_title: ASOS Does Not Meet Overall Reliability Specifications
paragraphs: The ASOS specification calls for mean time between system critical failuresand system critical errors to be 2,190 and 8,760 hours, respectively. NWS system reliability demonstrations during winter months in 1993 and 1994 indicated that ASOS was not meeting these requirements. The 1994 results show mean time between critical failures to be 1,200 hours (versus 2,190) and mean time between critical errors to be 2,800 hours (versus 8,760). Furthermore, critical failures and errors that were caught and corrected by weather observers were not included in these results. Once these errors are included, ASOS failure and error rates are 730 and 1,680 hours, respectively (see figure 2.1).
The ASOS program manager said that the January 1994 results are outdated, and that more recent performance data show that ASOS meets specifications with a mean time between failures on aviation-critical observations of 3,055 hours (versus the 2,190 hours requirement). This claim, however, is not supported by the facts. The data upon which the claim is based are incomplete and insufficient to calculate ASOS reliability. The specification defines system critical failures as the loss of any of the following observations: pressure, sky condition, visibility, wind, freezing rain, and precipitation. NWS’ recent performance data, however, do not include measurements taken during the harshest winter weather when ASOS reliability is poorest. Further, the data do not include failures by the precipitation identification and precipitation accumulation sensors. Therefore, these data alone are not indicative of ASOS’ overall reliability.
The reason that ASOS’ reliability problems were not discovered during testing and corrected prior to system deployment and operation is that ASOS program management repeatedly chose to defer testing of mean time between failures. Instead, the program office relied on the results of a model run by the contractor to predict system reliability, rather than testing reliability. After production, NWS’ testing branch finally performed a reliability test, but these tests did not include mean time between failures. Instead, it was decided that reliability could be better determined by analyzing actual maintenance actions on operational systems than by testing the preproduction system. While we agree that continuous monitoring of maintenance is important, it does not substitute for predeployment testing of performance specifications as practiced by leading information technology organizations. Further, it does not protect the government from buying production units that do not meet specifications for reliability.
Low system reliability increases the likelihood that wrong information will be disseminated to system users, again potentially risking aviation safety, impairing environmental research, and misguiding general public decision-making. Low reliability also results in a larger maintenance workload, lower user confidence in ASOS, and continued dependence on human observers for quality control. For example, one commissioned ASOS site reported that on occasion the ceilometer took 5 days to fix. During that time, a human observer had to take and disseminate all cloud height observations in order for airport operations to continue.
A senior AOMC manager stated that the planned modifications on individual sensors should improve ASOS’ overall reliability. While overall system reliability should improve as individual sensor improvements are implemented, there is no assurance at this time that the required reliability will be attained.
section_title: Requirements for Long Distance Quality Monitoring of FAA ASOS Units Not Being Met
paragraphs: Successfully communicating information between an ASOS site and the AOMC is critical to effective operation and maintenance of ASOS. AOMC monitors all ASOS sites for system-detected errors or failures, and dispatches technicians to fix problems. It is required to provide uninterrupted monitoring of ASOS observations 24 hours a day, 365 days per year. However, AOMC has not been able to provide this level of uninterrupted monitoring on FAA systems because an average of 16 percent of all observations transmitted from ASOS sites to the AOMC over FAA’s wide area network were lost in 1994. Comparatively, only about 1 percent of observations transmitted from ASOS sites to the AOMC over NWS’ network are lost. Although NWS has no documented quantitative criteria, the ASOS program manager stated that an acceptable percentage of lost messages would be less than 5 percent. NWS has refused to commission individual FAA sites until communications improve. The FAA ASOS program manager noted that the 16-percent data loss is high because it is an average of new and old ASOS sites. He stated that data communications at new sites are more problematic and therefore raise the average.
To monitor FAA ASOS sites, the AOMC must use FAA’s wide area network. NWS and FAA officials agree that a major reason for lost data is that the FAA network cannot support ASOS monitoring requirements in some locations. This network is configured such that a microwave signal must travel between as many as 30 points. This number of “hops” in the microwave transmission causes signal degradation or loss. FAA and NWS are working together to improve FAA communications through changes in the network configuration.
Because of this communication shortfall, AOMC cannot adequately monitor or respond to all ASOS problems at FAA sites. For example, an FAA ASOS site could be inoperative or provide inaccurate observations, and no maintenance technician would be dispatched, or maintenance technicians would be dispatched when none are needed. AOMC instituted a “work-around” in which it would have FAA troubleshoot the communications line when an hourly observation is missing, and then contact a maintenance person only if the next hourly observation is missing as well.
section_title: NWS Has Historically Been Effective in Resolving ASOS Shortfalls in Meeting Specifications
paragraphs: Successfully operating and maintaining a system requires an effective process for reporting, tracking, and resolving instances where the system does not meet specified requirements. Such a process includes providing users with a vehicle for easily and quickly documenting and transmitting their concerns to a control point that (1) centrally records all problems, (2) controls disposition status of these problems, and (3) provides feedback to the problem originator on disposition actions. This process also includes active participation by the system program office, contractor, and change control board in evaluating the nature of the problem and deciding on corrective actions, if any.
The NWS program office has an effective process for reporting ASOS problems and resolving those problems it finds to be shortfalls in meeting specifications. It provides users with a standard form, commonly called a trouble report, for them to easily and quickly document their concerns and transmit them to the program office. The program office centrally records these trouble reports, tracks their disposition status, and provides feedback to the problem originator on disposition actions. Trouble reports are evaluated by the program office, contractor, and change control board to determine if the problem results from (1) a shortfall in meeting specifications, (2) a defective system component in need of maintenance, or (3) a misunderstanding of the system’s functions highlighting an issue that is beyond the specification. Trouble reports are closed when the contractor fixes the shortfall, the needed maintenance is provided, or the program office responds to the trouble report originator explaining that the problem identified did not constitute a specification shortfall.
Through its trouble reporting process, the ASOS program office has established a good record in resolving ASOS problems that it determined to be shortfalls in meeting specifications. Between July 1991 and December 23, 1994, the program office received a total of 575 trouble reports and, as of December 1994, had closed 470 of these. Of these 470, the program office reported that shortfalls in meeting the specification were fixed in 224 cases, the program office provided needed maintenance in 22 cases, and the program office responded to the trouble report originator that the problem identified did not constitute a contract shortfall on 148 cases. The remaining 76 cases included duplications of prior trouble reports and reports that the originator withdrew.
Early in our review, we sampled the database of trouble reports to determine NWS’ effectiveness in resolving user reports of ASOS failing to meet specified requirements. On the basis of our sample data, we project that about 96 percent of the full population of reported problems in meeting the ASOS specification are being rectified to the satisfaction of their originators. This, however, does not address user satisfaction with NWS resolution of ASOS problems deemed to be beyond the scope of the specification. This question is addressed in chapter 3 of this report.
section_title: NWS Beginning to Assess Needs That Users Have Long Claimed ASOS, as Specified, Does Not Satisfy
paragraphs: ASOS user groups have repeatedly cited needs that the system, as specified, does not meet. While NWS has recently started assessing these user concerns and plans to decide whether or not to enhance or supplement the system to address them, NWS’ track record in effectively resolving ASOS reported problems that it deems to be outside the scope of the specification is poor. Until it either enhances ASOS or otherwise supplements ASOS to address user concerns, NWS is giving its users a system that falls short of their expectations, cannot by itself replace human observers, and, according to these users, could result in poor weather-related decision-making.
subsections: section_title: NWS Has Yet to Decide How to Address User Concerns That ASOS, as Specified, Does Not Satisfy
paragraphs: In our 1994 report on how leading organizations improved mission performance through strategic management of information and technology, we reported that the success and value of a system is largely measured by the extent to which it meets users’ needs.
The primary users of ASOS are the aviation, meteorology, climatology, and hydrology communities. Each relies on weather observations that have long been provided by human observers. According to these users, however, ASOS as specified does not fully substitute for the human observer and, therefore, does not meet their needs. We did not attempt to determine the validity and criticality of these unmet needs. However, ASOS users stated that these needs are important and not meeting them could cause adverse consequences.
NWS is now identifying, evaluating, and, in some cases, resolving ASOS users’ unmet needs. It is holding workshops with FAA and aviation organizations to further define requirements, and it is documenting climatologists’ concerns. Additionally, because the ASOS display equipment originally specified in the ASOS contract for use in airport towers did not satisfy air traffic controllers’ needs, NWS has modified the specification and the contractor is now developing new display equipment.
NWS officials stated that not all user needs can be allocated equal weight and urgency. NWS’ top priorities are life and safety issues, such as supporting aviation operations and providing public warning and forecast services. However, NWS officials also stated that some user communities, such as climatological information users, could benefit from additional observations and data and that NOAA will work to establish requirements and formulate and justify cost-effective means of meeting them. The ASOS program, for example, is to explore development of sensors to obtain frequently requested data that ASOS does not now provide. Additionally, the National Research Council is proposing a new committee to assess the costs and benefits of climate data needs and recommend strategies for fulfilling these needs. Using these inputs, NWS plans to define a comprehensive set of climatological requirements during the next 2 years.
NWS officials also noted that the ASOS program has suffered from changing FAA requirements. For example, they stated that FAA’s requirement for tower display equipment has changed significantly and often, resulting in several previous tower display designs. Additionally, they stated that FAA’s operational availability requirements have changed and have yet to be adequately defined.
subsections: section_title: Aviation Community Users State That ASOS Does Not Satisfy Need for Representative Observations
paragraphs: Representative observations are weather observations that accurately describe the actual, prevailing conditions for the airport area. Aviation users stated that it is critical that visibility and cloud ceiling observations be representative. According to officials from NATCA, the Airline Dispatchers Federation (ADF), and the Air Transport Association (ATA), if visibility and ceiling observations are not representative, inappropriate flight rules can be applied, causing inefficient use of airspace or placing unnecessary constraints or risks on aircraft operations. Specifically, when reported observations are much worse than actual weather conditions, pilots may unnecessarily avoid the airport, or be required to undertake a more complex and costly instrument-guided approach. Alternatively, when reported observations are much better than actual weather conditions, pilots could decide to use an airport when they otherwise would not.
NATCA reported in October 1994 that ASOS readings are unrepresentative when the weather is changing rapidly or conditions are patchy. This has led to situations, for example, in which aircraft were routed for an instrument-guided approach because ASOS was reporting low visibility, but the pilot could see the runways on approach from miles away. Also, one controller described an incident in which several small aircraft were allowed to take off based on the ASOS report of 2,500-foot cloud ceilings. A pilot radioed in from the air to say that ceilings were actually at 900 feet. According to the controller, this variance could put small aircraft at risk.
The ASOS specification does not require that its observations be representative of actual, prevailing weather conditions. In fact, the specification never mentions representativeness. Instead, the specification requires ASOS to perform a time average of weather within a small area (i.e., a sample area) rather than a spatial average as a human would when scanning the entire sky. Thus, when weather is changing rapidly or is patchy, ASOS observations can be accurate for the small area it is viewing, but unrepresentative of prevailing weather conditions in the universe being sampled. According to NWS and FAA program managers, ASOS is not, and never was, intended to replicate the way a human observes weather conditions, and that the aviation community must undergo a “paradigm shift” to accept automated observations.
Neither NWS nor FAA has reliable data on the extent to which ASOS observations are unrepresentative or the effect of unrepresentative observations on aircraft operations. A senior NWS manager stated that such data have not been systematically collected and analyzed because to date both NWS and FAA management have focused on correcting ASOS where it does not meet specifications rather than enhancing the system to meet users’ needs. Some limited data exist, which although not conclusive, support the users’ statements that unrepresentativeness is a problem. For example, a New York weather office recorded an average of 51 unrepresentative events per month from November 1993 to February 1994 and an average of 32 events per month from March to May 1994. These data are discussed in appendix II.
The ASOS program office is drafting a plan for systematically collecting and assessing data on unrepresentative ASOS observations, after which it will decide whether or not to enhance or supplement the system to address this concern. At the same time, NWS is also collecting and assessing other aviation requirements. Through a series of workshops with FAA, NATCA, and aviation industry officials, NWS is identifying these users’ minimum service requirements and evaluating plans for enhancing and supplementing ASOS to satisfy any unmet needs.
section_title: Meteorologists Cite Several Needs Beyond the ASOS Specification That Are Not Being Met
paragraphs: NWS meteorologists at the Sterling Research and Development Center identified three areas in which ASOS does not provide observations that they describe as significant to weather reporting and that human observers currently provide. These areas are (1) identifying ice pellets as solid precipitation, (2) correctly identifying rain, snow, or no precipitation more frequently than currently specified in ASOS’ specification, and (3) distinguishing between actual cloud layers and invisible layers of particles in the atmosphere. According to ASOS users, because this information is particularly important to aviation forecasts, incorrect observations could risk aviation safety, inconvenience airport operations, and add to aviation costs. NWS is planning to invest resources in ASOS enhancements to address two of these three unmet needs. Each unmet need is described below.
The ASOS specification states that solid precipitation shall be correctly detected at least 99 percent of the time (reported as either snow or mixed/other precipitation) and shall be correctly identified at least 97 percent of the time. According to the ASOS Technical Chief, solid precipitation should mean only snow. It should not mean other forms of solid precipitation, such as ice pellets.
ASOS satisfies the program office’s narrow definition of solid precipitation. However, ASOS tests and operational experience show that the system misidentifies solid precipitation other than snow (e.g., it often identifies ice pellets as rain). According to ASOS users, failure to correctly identify other types of solid precipitation could pose a safety problem should the system report rain in ice pellets, hail, or icing conditions, and the problem is exacerbated at unstaffed ASOS sites where human observers would not be present to correct erroneous reports. The error occurs because the sensor identifies precipitation types on the basis of their size and velocity. The ASOS program office is evaluating alternative technological enhancements that could permit discriminating ice pellets and hail from other forms of solid precipitation.
The ASOS specification requires that the precipitation identification sensor correctly detect precipitation (liquid and solid) at least 99 percent of the time, and correctly identify at least 90 percent of liquid precipitation and 97 percent of solid precipitation. According to the ASOS program office, ASOS meets this specification. However, NWS’ Office of Systems Operations, which is responsible for the operation and maintenance of commissioned units, reported that current performance levels may not satisfy aviation operational requirements. As stated by a senior testing manager, 1 percent of 24 hours of precipitation is about 15 minutes, and if ASOS does not correctly detect precipitation for 15 minutes while an aircraft is approaching, a significant safety risk could be incurred. The significance of the situation was underscored in the workshops involving NWS, FAA, and aviation industry officials to define aviation industry requirements for weather information. During these workshops, industry officials stressed the importance of having more accurate and available precipitation information. To address this concern, the ASOS program office is assessing technical enhancements to improve this sensor’s accuracy.
ASOS users cite requirements for accurately identifying cloud ceilings. Because of technological limitations, ceilometers alone (i.e., without human observers) cannot currently provide this. ASOS ceilometers falsely identify weather phenomena such as invisible particles, moist air layers, virga, and precipitation as cloud layers. This can result in observations of broken or overcast ceilings when in fact there are clear or scattered conditions. Broken and overcast conditions are defined by the Federal Aviation Regulations as a ceiling, and ceilings determine legal limits and flight rules for various aircraft operations.
NWS managers stated that this problem does not risk aviation safety; rather, it inconveniences the aviation community. Moreover, they stated that it occurs too infrequently to be a concern. In contrast, aviation industry officials stated that even infrequent occurrences can be costly to airlines. Despite this, program officials stated that they have no plans for researching and developing new technology to address the concern.
section_title: Climatologists Contend That ASOS Is Not Providing Needed Precipitation Observations
paragraphs: Officials from the National Climate Data Center, the American Association of State Climatologists, and the ASOS Climate Working Group, as well as various prominent climate researchers from industry and academia, stated that climatologists need uninterrupted and accurate measurement of precipitation. However, ASOS does not meet this need. In February 1994, NOAA’s Climate Data Continuity Project reported that ASOS’ precipitation accumulation sensor generally reports less precipitation than conventional sensors, with the greatest underreporting occurring in summer and winter months. The report explained that ASOS’ gauge performed reasonably well in rain events that were of light to moderate intensity but did not perform well in snow or freezing conditions or in heavy rain events that are typical in the summer. The project further reported that, from the summer of 1993 through the spring of 1994, ASOS precipitation observations were 94 and 91 percent of conventional observations in fall and spring, respectively, but fell to 86 percent in summer and only 75 percent in winter (see figure 3.1).
Climatologists stated that incorrect precipitation observations affect not only weather-related decision-making, but also accurate climatological analysis. Specifically, inaccurate precipitation records could affect (1) businesses that rely on this information for everyday decisions, (2) climatological analysis of flood and drought frequencies, intensities, and durations, and (3) long-term national climate research on global and regional precipitation, potentially skewing national environmental policy decisions on global change. To address this climatological need, NWS is developing specifications and plans to procure an all-weather precipitation gauge to better report liquid and frozen precipitation amounts.
According to the ASOS program manager, ASOS does not meet all climatology needs because it was not originally intended to do so. Rather, it was intended only to meet forecasting and aviation requirements. The program office, however, is now assessing how best to meet climatologists’ requirements. It has started meeting with climatology users to identify their unmet needs and concerns, including how to supplement ASOS to ensure that needed observations, which are currently provided by the human observers, continue once the human observers are removed.
section_title: ASOS Airport Tower Display Equipment Does Not Meet FAA’s Needs
paragraphs: The ASOS specification defines the display equipment needed by air traffic controllers in airport towers. Originally, the specification did not require that the ASOS display equipment integrate data from other airport tower displays. However, after operational experience in the towers with the ASOS displays, FAA raised several concerns, including that the ASOS displays were forcing controllers to view too many tower displays. As a result, FAA and NWS changed the ASOS specification to require enhanced displays that could integrate data from other tower displays with the ASOS data. Currently, FAA is testing the new, integrated display equipment in several towers. If testing is successful, FAA plans to accept and deploy the equipment. An FAA official estimated that the redesigned displays will cost between $10 and $12 million, depending on the number bought.
section_title: NWS’ Process for Resolving Problems Is Not Effective for Concerns Deemed Outside the Scope of the Specification
paragraphs: Successfully deploying, operating, and maintaining a system requires an effective process for resolving user problems. As discussed in chapter 2, NWS’ process for reporting and resolving problems where the system fails to meet specified requirements is effective. However, NWS does not have an effective process for resolving user concerns that are beyond the scope of the ASOS specification.
Internal NWS memoranda and NWS managers described the ASOS problem resolution system as unresponsive to system users whose problems were determined to be outside the scope of contract specifications. To validate these statements, we randomly sampled 50 trouble reports and found  20 reports that the program office closed because it found the system’s reported performance to be consistent with specified requirements. The originators on 11 of these 20, however, were not satisfied with the program office’s actions. For example, one technician reported that the size of the openings in the mesh screen on the temperature and dew point sensor was too large, allowing too many bugs to enter and thus requiring the sensor to be cleaned as much as 12 times more frequently than the maintenance manual required. The ASOS program office closed this trouble report on the basis of the mesh screen size being compliant with the size stated in the contract specification.
The ASOS program officials stressed that their priority has been in getting the system to meet specifications. They described other user requirements as being of lesser importance. ASOS program officials also stated that the trouble report process was only intended to handle shortfalls in specifications and that other processes should be followed to seek changes or enhancements to the system. They acknowledged, however, that users often are unable to distinguish between specification shortfalls and system enhancements and thus will often file trouble reports to surface their concerns. In cases where the reported problem is not a specification shortfall, the user is told to file a different report to request a system enhancement. However, this approach places the burden on the user to resubmit a concern already raised. Without a user-friendly problem reporting process, valid concerns may not be surfaced, increasing the possibility of NWS fielding a system that fails to satisfy valid user requirements.
To improve its record in resolving all trouble reports, the ASOS program office set up a new quality assurance working organization that will review all trouble reports proposed for closure. The ASOS program manager stated that this group will also be more responsive to user concerns that are beyond the specification.
section_title: Cost of Addressing All ASOS Reported Problems Is Not Known
paragraphs: Because NWS is still assessing many of the problems facing ASOS, it has yet to estimate the cost to resolve them, either via system enhancements or human/system supplementation. Without reliable estimates of what an enhanced or supplemented ASOS that fully meets users’ needs will ultimately cost, NWS does not know whether continued investment in ASOS is cost beneficial.
The Federal Acquisition Regulation requires that the government develop an estimate of the cost of any goods and services to be acquired. Further, the Federal Information Resources Management Regulation requires that a system’s life cycle costs (i.e., the cost to design, develop, deploy, operate, and maintain a system over its expected useful life) be used in any analysis of the relative costs and benefits of competing system alternatives. Cost estimates are important considerations in all investment decisions related to the goods and services being acquired, such as whether the benefits expected from the goods and services outweigh their costs and thus whether investment in these items is justified. Further, these cost estimates should not be static. They are to be appropriately refined and updated throughout the acquisition cycle.
NWS does not have a complete estimate of what it will cost to address all user concerns with ASOS. The current cost estimate of $351 million includes the cost to purchase and deploy 868 ASOS units as contractually specified. Under the ASOS firm, fixed-price contract, the contractor is to absorb all costs to fix shortfalls in meeting these contract specifications. However, the government is responsible for (1) correcting any shortfalls in meeting specified requirements that are due to deficient government-furnished sensors, (2) improving or upgrading the system beyond its specified requirements, and (3) supplementing the system with observations from other sources (e.g., humans or other systems).
The cost of those problems that are due to deficient government-furnished sensors, such as the dew point sensor and the ceilometer, are included in the $351 million cost estimate. However, other reported problems or concerns that are outside the scope of the ASOS specification are only now being assessed, and thus the total cost to address them, either by improving ASOS or supplementing it, is unknown.
According to NWS officials, the ASOS 1995 and 1996 budgets include $3.9 million for product improvement activities to correct current sensor limitations and meet remaining commonly accepted requirements. These requirements include a sunshine sensor, an all-weather precipitation accumulation sensor, an ice-free wind sensor, a new dew point sensor, and a sleet and hail detection sensor. While the cost to produce the sensors is not yet known, NWS officials estimate that production costs will not exceed $22 million for the 868 ASOS units in the base program.
Supplemental costs associated with residual backup requirements at major airports are not yet known because NWS, FAA, and various aviation organizations are currently negotiating the required level of backup. NWS officials stated that if a very high availability is required, human observers would not be released, and thus expected cost reductions would not be achieved. NWS officials estimated that if they are unable to release observers, the cost of human augmentation and backup in NWS alone will be $16 million for 336 staff years in fiscal year 1996, eventually rising to approximately $33 million for 615 staff years per year.
section_title: Plans for Implementing ASOS Lack Effective User Involvement
paragraphs: Before relying on ASOS to provide NWS’ official weather observations, NWS should demonstrate that all valid user needs are being met. To date, this has not occurred. Accordingly, NWS recently discontinued ASOS commissionings and postponed plans for releasing its human weather observers until ASOS problems are addressed. However, NWS has not made plans to ensure that all users are effectively involved in decisions on releasing human observers.
NWS expects ASOS to replace NWS and contract weather observers at about 175 weather offices and airports, and FAA plans to release its contract observers at 194 sites. Before releasing any observers, however, NWS must ensure that either ASOS or some supplementary observing source can meet the needs of ASOS’ different user communities.
While NWS plans for ensuring that ASOS meets user needs effectively involve ASOS’ aviation users, they do not ensure that other ASOS users will be adequately involved. Specifically, NWS has postponed plans to release observers while it (1) reaches agreement with the aviation community as to what weather information is essential for aviation operations, (2) determines, through a 6-month demonstration, the workload associated with augmenting ASOS observations with information ASOS cannot provide (e.g., tornadoes, thunderstorms, hail, volcanic ash, and virga) and supplementing ASOS observations in case ASOS fails or errs, and (3) arranges to have non-NWS staff augment and supplement ASOS as needed, based on demonstration results. These efforts, however, do not address unmet needs that other ASOS users have identified as important.
The extent to which other ASOS users will be involved in decisions regarding the release of human weather observers is not clear. According to the ASOS program manager, human observers will not be released until all users’ needs are met either through ASOS or some supplementary source. However, he acknowledged that this is not confirmed in any documented plan or strategy for human observer release. Further, it contradicts the explicit purpose of NWS’ demonstration for aviation users, which links human observer release to meeting solely aviation user needs. Without a plan for releasing human observers that includes explicit criteria reflecting the needs of all ASOS users, NWS runs the risk of releasing observers before adequate supplementary sources are in place.
section_title: Conclusions, Recommendations, and Agency Comments and Our Evaluation
subsections: section_title: Conclusions
paragraphs: ASOS is performing neither as intended nor expected. While many specified requirements are being met and steps are underway to correct instances where they are not, the system does not provide certain information that human observers do and users say is important to aviation safety, weather-related decision-making, and climatological analysis.
Whether an enhanced ASOS can eventually provide the information users say they need or whether ASOS will need to be supplemented by other observing technology or people is not yet known. What is known is that ASOS problems became severe enough for NWS and FAA to temporarily halt commissioning additional ASOS units and to refrain from implementing plans to release human observers until the problems are solved. However, NWS has yet to determine what specific ASOS problems must be solved before observers can be released, what the solutions will cost, and whether expected ASOS benefits will be worth the additional costs. Until NWS works collaboratively with ASOS users in doing so, it runs the risk of deploying a system that is not cost-effective, and that does not adequately support important public safety and environmental decisions.
section_title: Recommendations
paragraphs: Given the criticality of accurate and timely weather observations, we recommend that the Secretary of Commerce direct the NOAA Assistant Administrator for Weather Services to define and prioritize, in conjunction with ASOS’ users, all system corrections, enhancements, and supplements that must occur to meet valid user needs; reestimate ASOS’ costs in light of planned system corrections, enhancements, and supplements; formulate, in conjunction with ASOS users, explicit system performance and cost/benefit criteria governing the release of human observers; and certify to the Secretary that (1) the cost/benefit criteria have been satisfied before investing in ASOS corrections, enhancements, and supplements and (2) the system performance criteria have been satisfied before releasing human observers.
section_title: Agency Comments and Our Evaluation
paragraphs: NOAA and FAA officials provided oral comments on a draft of this report. These officials generally concurred with the report’s findings. NWS officials emphasized that they are tackling ASOS’ technical problems and plan to identify, validate, and address outstanding user needs. Additionally, they noted that not all ASOS user needs can be allocated equal weight and urgency. They also stated that ASOS will satisfy aviation requirements, which they described as the most challenging, and that the essential requirements for ASOS’ contribution to other NWS operational services will be met. NWS officials added that other users’ needs will be satisfied through a combination of ASOS product improvements and other data sources.
These statements are not inconsistent with our report. However, the point remains that ASOS problems in meeting both specified requirements and user needs beyond the specification’s scope have yet to be resolved. Moreover, what it will take in terms of the time and money needed to resolve these problems and when ASOS will ultimately permit the release of human observers remain uncertain. Until NWS addresses these uncertainties in collaboration with ASOS users, it does not know whether ASOS corrections, enhancements, and supplements are worth the investment.