Aviation Weather: Services at Key Aviation Facilities Lack Performance Measures, but Improvement Efforts Are Under Way
Released: 2008-02-26T00:00:00

Letter

Background

FAA is responsible for ensuring safe, orderly, and efficient air travel in the national airspace system. NWS supports FAA by providing aviation-related forecasts and warnings at air traffic facilities across the country. Among other support and services, NWS provides four meteorologists at each of FAA’s 21 en route centers to provide on-site aviation weather services. This arrangement is defined and funded under an interagency agreement.

FAA’s Mission and Organizational Structure

FAA’s primary mission is to ensure safe, orderly, and efficient air travel in the national airspace system. The agency’s ability to fulfill its mission depends on the adequacy and reliability of its air traffic control systems, as well as weather forecasts made available by NWS and automated systems. These resources reside at, or are associated with, several types of facilities: air traffic control towers, terminal radar approach control facilities, air route traffic control centers (en route centers), and the Air Traffic Control System Command Center. The number and functions of these facilities are as follows: 517 air traffic control towers manage and control the airspace within about 5 miles of an airport. They control departures and landings, as well as ground operations on airport taxiways and runways. 170 terminal radar approach control facilities provide air traffic control services for airspace within approximately 40 miles of an airport and generally up to 10,000 feet above the airport, where en route centers’ control begins. Terminal controllers establish and maintain the sequence and separation of aircraft. 21 en route centers control planes over the United States—in transit and during approaches to some airports. Each center handles a different region of airspace. En route centers operate the computer suite that processes radar surveillance and flight planning data, reformats them for presentation purposes, and sends them to display equipment that is used by controllers to track aircraft. The centers control the switching of voice communications between aircraft and the center, as well as between the center and other air traffic control facilities. Two en route centers also control air traffic over the oceans.

The Air Traffic Control System Command Center manages the flow of air traffic within the United States. This facility regulates air traffic when weather, equipment, runway closures, or other conditions place stress on the national airspace system. In these instances, traffic management specialists at the command center take action to modify traffic demands in order to keep traffic within system capacity.

See figure 1 for a visual summary of the facilities that control and manage air traffic over the United States.

NWS’s Mission and Organizational Structure

The mission of NWS—an agency within the Department of Commerce’s NOAA—is to provide weather, water, and climate forecasts and warnings for the United States, its territories, and its adjacent waters and oceans to protect life and property and to enhance the national economy. In addition, NWS is the official source of aviation- and marine-related weather forecasts and warnings, as well as warnings about life-threatening weather situations.

The coordinated activities of weather facilities throughout the United States allow NWS to deliver a broad spectrum of climate, weather, water, and space weather services in support of its mission. These facilities include 122 weather forecast offices located across the country that provide a wide variety of weather, water, and climate services for their local county warning areas, including advisories, warnings, and forecasts; 9 national prediction centers that provide nationwide computer modeling to all NWS field offices; and 21 center weather service units that are located at FAA en route centers across the nation and provide meteorological support to air traffic controllers.

NWS Provides Aviation Weather Services to FAA

As an official source of aviation weather forecasts and warnings, several NWS facilities provide aviation weather products and services to the FAA and aviation sector. These facilities include the aviation weather center, weather forecast offices located across the country, and center weather service units located at FAA en route centers. See table 1.

Center Weather Service Units: An Overview of Current Requirements

FAA’s existing requirements for the center weather service units are broadly outlined in an interagency agreement that is updated every few years. The interagency agreement specifies that NWS is to provide meteorological advice and consultation to en route center operations personnel and other designated FAA air traffic facilities within the en route area of responsibility. This agreement establishes specific terms that govern the number of NWS staff, their working hours, and cost reimbursement details. It does not specify the contents, quality, or frequency of weather products.

An NWS directive, signed in May 2006 and intended for NWS’s weather forecast offices and center weather service units, provides more specific information regarding the content of weather products and services, including center weather advisories, daily briefings, on-demand consultations, and meteorological impact statements. These products and services are described in table 2. In addition, center weather service unit meteorologists can provide input every 2 hours to the Aviation Weather Center’s creation of the Collaborative Convective Forecast Product; train FAA personnel on how to interpret weather information; and, if warranted, provide weather briefings to nearby terminal radar approach control facilities.

FAA Sought to Improve Aviation Weather Services Provided at En Route Centers

In recent years, FAA has sought to assess and improve the performance of the center weather service units. For example, FAA performed multiple studies on the current services provided by the center weather service units that noted the lack of standardization of products and services. In addition, FAA conducted a study to determine if remote operations were feasible, and requested that NWS restructure its aviation weather services to provide improved services more efficiently. In response to this request, NWS conducted a prototype of remote operations in which center weather service unit products and services were prepared by the closest weather forecast office. NWS proposed expanding this prototype to FAA, but the agency declined this proposal. Instead, FAA stated that it would redefine its requirements for the functions provided by center weather service units. Table 3 provides more information about the agency’s assessment and improvement efforts.

FAA Found Its Requirements Were Not Sufficiently Precise and Worked to Develop New Ones

When FAA declined NWS’s proposal for restructured aviation weather services, it did so in part because it considered its existing requirements governing NWS’s center weather service units to be too broad to ensure the efficiency and cost-effectiveness of the services. FAA then worked for several months to redefine these requirements. In April 2007, FAA’s Air Traffic Organization began refining its requirements for aviation weather services at the en route centers. To do this, FAA collected relevant NWS and FAA orders and directives and developed a list of over 100 products and services that the different service units provide. FAA then sent this list to traffic managers in each of the en route centers, asking them to specify the products and services that they need, the ones they do not need, and any new products or services that they would like. Traffic managers were also able to determine if they would want some of the more customized weather products that are currently available at selected en route centers. Using results from this survey, FAA developed a revised list of requirements and performance measures, which it provided to NWS in late December 2007.

Neither NWS nor FAA Currently Ensures the Quality of Aviation Weather Services at En Route Centers

While interagency agreements between NWS and FAA state that both agencies have responsibilities for assuring and controlling the quality of aviation weather observations, neither NWS nor FAA consistently does so for weather products produced at the en route centers. Leading organizations use quality assurance to provide staff and management with objective insights into processes and associated work products. Generally, quality assurance includes objectively evaluating performed processes, work products, and services against applicable process descriptions, standards, and procedures; identifying and documenting noncompliance issues; providing feedback to project staff and managers on the results of quality assurance activities; and ensuring that noncompliance issues are addressed. However, neither NWS nor FAA has developed and implemented performance measures and metrics, regularly evaluated weather service unit performance, or provided feedback to improve these aviation weather products and services.

Because of this lack of performance tracking and oversight, NWS cannot demonstrate the quality or value of its services, and FAA cannot ensure the value of the services it funds. As a result, it is not clear that FAA is getting the information it needs to effectively manage air travel. FAA officials stated that they intend to establish performance measures for their redefined requirements and to improve their oversight against these measures. However, FAA has not worked with NWS to define a comprehensive set of measures for its requirements, and it is unclear how the agency would develop a performance baseline for comparison to actual performance because many of the products and services have not previously been measured.

NWS Does Not Measure or Evaluate Aviation Weather Products and Services at En Route Centers

NWS does not measure or evaluate the aviation weather services it provides at en route centers. Under existing interagency agreements, NWS is responsible for controlling the quality of its aviation weather observations. Specifically, the agency is responsible for monitoring and evaluating the quality and effectiveness of its aviation weather services, including the services provided at the weather forecast offices, the Aviation Weather Center, and the en route centers.

While NWS has developed and continues to monitor performance measures for aviation weather forecasts provided by its weather forecast offices and the Aviation Weather Center, the agency has not done so for the weather products and services provided at the en route centers. Specifically, NWS has not developed performance measures for aviation weather products and services at en route centers, evaluated the aviation weather products and services developed at the en route centers, or provided feedback for those services. NOAA and NWS officials declined to explain why the agency does not have performance measures for aviation weather products or services at en route centers, but they noted that neither FAA nor NWS has required or funded such an effort. Further, the aviation services branch chief told us that he had planned to begin evaluations for weather unit services at the en route centers but decided to wait because of the potential for large-scale changes to the services.

Until NWS establishes performance measures and evaluates the quality and effectiveness of its products against these measures, the agency will remain unable to ensure that it provides consistent quality products and to effectively demonstrate the value it provides to FAA.

FAA Does Not Consistently Evaluate or Provide Feedback on Aviation Weather Services at En Route Centers

FAA has not consistently evaluated NWS services at its en route centers or adequately provided feedback on the results of its few evaluations. Under interagency agreements, FAA is responsible for ensuring that aviation weather services meet its requirements. In addition, it requires the traffic management officer within each traffic management unit to evaluate the aviation weather services at the en route centers annually and to provide feedback to the resident meteorologist-in-charge.

FAA has not consistently ensured the quality of aviation weather services at en route centers. Specifically, it currently does not have any quantitative and objective performance measures—such as timeliness, accuracy, or false alarm rate—by which to evaluate these services. Agreements between the agencies broadly specify the types of aviation weather products to be developed at the en route centers but do not provide criteria by which these products can be evaluated. In addition, FAA has not consistently performed its annual evaluations of these products and services. According to the contracting officer’s technical representative responsible for the evaluations, the last evaluation was performed in 2006, and its results were largely anecdotal. Specifically, the evaluation called for the traffic management officer to rate the weather unit on a scale of 0 to 4 in different categories, including quality and timeliness of products and services, knowledge of air traffic control, and participation in training.

The technical representative could not find any evaluations in 2005, evaluations of only three service units in 2004, and evaluations of a similarly small number of service units in 2003.

Further, FAA is not consistently providing feedback to weather staff at the en route centers. According to the technical representative, the evaluations from 2006 were not compiled or analyzed because the evaluations contained no glaring problems or issues that needed additional attention. In addition, the NWS aviation services chief told us that FAA had sent him copies of the evaluations from 2006 but did not offer analysis of these evaluations, express concerns about the services, or send the results to the individual center weather service units. This official also stated that he was not aware that FAA had performed any annual evaluations of the center weather service units prior to 2006.

Because FAA has not established performance requirements or consistently and thoroughly evaluated the aviation weather services at en route centers, the agency cannot be sure that the products and services provided by the weather unit meteorologists are adding value, and it cannot provide feedback to NWS meteorologists in order to improve the services. To address this shortfall, FAA officials stated that they intend to establish performance measures for aviation weather services at en route centers when they revise their requirements and to improve their oversight of NWS against these measures. However, FAA has not worked with NWS to develop measures for the products and services it will require from NWS, and it is unclear how the agency would develop a performance baseline for comparison to actual performance because many of the products and services have not previously been measured.

Implementation of GAO Recommendations Should Help Address Performance Measurement Shortfalls

While many steps remain in defining the future of aviation weather services at en route centers—including negotiations between FAA and NWS on the provision of these services and FAA’s subsequent decision on whether to obtain selected services from alternative sources—there are steps both agencies can take to ensure that the quality of future aviation weather products and services is measured and evaluated. In our accompanying report released today, we made two recommendations to the Secretary of Commerce and three recommendations to the Secretary of Transportation to improve the quality of aviation weather products and services at en route centers.

We recommended that the Secretary of Commerce direct the Assistant Administrator for the National Weather Service to assist FAA in developing performance measures and metrics for the products and services to be provided by center weather service units, and perform annual evaluations of aviation weather services provided at en route centers and provide feedback to the center weather service units.

Further, we recommended that the Secretary of the Department of Transportation direct the FAA Administrator to work with NWS to define performance measures and metrics for aviation weather services provided by meteorologists, evaluate the services it receives against those measures and metrics, ensure that results of these evaluations are provided to staff stationed at center weather service units so that they can improve performance, where applicable.

In written comments on a draft of our report, the Secretary of Commerce agreed with our recommendations to assist FAA in developing performance measures and metrics, and to perform annual evaluations of aviation weather services and provide feedback to the center weather service units. The department stated that after FAA provides its revised requirements, it would work with FAA to develop methods for performance monitoring and evaluation. Subsequently, on December 19, 2007, FAA provided its revised requirements to NWS.

The Department of Transportation’s Director of Audit Relations provided comments on a draft of the report via e-mail. In those comments, the department did not agree or disagree with our recommendations. The department stated that FAA’s revised requirements are consistent with our recommendations in that they establish performance measures and evaluation procedures, and that FAA would begin to negotiate with NWS to implement them. In addition, in late December 2007, after reviewing our draft report, FAA and NWS signed a new interagency agreement that requires FAA to develop performance standards and measures for the assessment of center weather service units, and requires NWS to develop and track metrics to support FAA’s performance measures.

FAA Identified New Aviation Weather Requirements and Performance Measures

FAA has already begun to address our recommendations; specifically, in late December 2007, FAA finalized its new requirements for the aviation weather services to be provided by center weather service units, which include proposed performance measures and methods for evaluation. In its requirements, FAA provides NWS with its overall vision for aviation weather services, revises existing requirements, and defines a requirement for a new product for terminal radar approach control facilities. In addition, FAA identifies performance measures and processes for evaluating performance and providing feedback to NWS.

FAA envisions services that are performance-based, standardized, continuous, and have a national scope. Specifically, FAA requires performance-based services that are measurable and allow for identifying both successful performance and any performance problems. In addition, FAA requests that the center weather service units provide standardized services to all en route centers and increase their service coverage from 16 hours a day to 24 hours a day. Finally, FAA calls for transitioning the scope of the center weather service units to monitor the entire national airspace system, rather than the respective en route center regions. This national scope is expected to allow more integrated decision making at the national level while continuing to provide specialized products at the regional and local levels.

In its new requirements, FAA also reiterates its need for existing products and services and provides revisions to some of these requirements. Specifically, FAA continues to require products such as twice-daily briefings, center weather advisories, and the Collaborative Convective Forecast Product. In addition, center weather service units will continue to provide forecast coordination with other NWS offices, on-demand advice and consultation, emergency planning, training, and dissemination of a number of weather advisories into both NWS and FAA systems. Daily briefings, however, will now be recorded, verified, and disseminated to other facilities that do not receive an in-person briefing. In addition, on- demand consultation will be provided to en route centers, terminal radar approach control facilities, towers, and the Air Traffic System Command Center. According to the aviation services branch chief, this consultation is currently provided only to en route centers, selected terminal radar approach control facilities, and a small number of towers.

Further, the revised requirements define a new product: the terminal radar approach control forecast product. This product is based in part on decision aids currently used in select center weather service units, and on requirements developed by a team consisting of aviation meteorological stakeholders from industry and FAA. This forecast, which describes the next 6 hours and is updated at least every 2 hours, will be presented in a graphical format and include convection, winds, ceilings, and visibilities for the area around terminal radar approach control facilities. FAA also expects this product to include methods for verification and the systematic collection of user feedback.

In addition to these requirements, FAA identifies performance measures as well as processes for evaluating performance and providing feedback to the forecasters. These performance measures include customer satisfaction, forecast accuracy, and the aggregate of aircraft incidents attributed to inaccurate aviation weather forecasts. Baselines for all of these measures have not yet been developed. According to the chief of the aviation services branch, NWS will propose additional performance measures and develop baselines as it is able. To measure against these performance measures, FAA has identified methods by which to evaluate NWS. For example, to determine customer satisfaction, FAA plans to develop a questionnaire for traffic management unit staff to be filled out quarterly; to determine the aggregate of aircraft incidents attributed to inaccurate aviation weather forecasts, FAA is to use safety statistics currently tracked by the Air Traffic Organization. In addition, FAA is planning to draw on NWS’s subject matter expertise to record and analyze information to determine the accuracy of forecasts.

FAA expects NWS to respond as to whether it is able to meet the requirements by early May 2008. In addition, FAA directed NWS to include plans for three operational concepts (including technical and cost information) for fulfilling the requirements—in its existing configuration located at the 21 en route centers, through remote services provided by a reduced number of regional facilities, and through remote services provided by a single centralized facility. According to the requirements, NWS’s response should assume a transition time of 90 days for the existing configuration, 180 days for regionalized remote services, and 1 year for a single facility. For each of these concepts, FAA has requested that NWS include a technical plan, a facilities plan, a quality management plan, and a plan for transitioning to the new approach. In addition, FAA asked that NWS include cost plans for each of the concepts with the assumption of 1 base year and four 1-year options thereafter. The cost plan is also expected to include estimated annual cost savings over this 5-year period. FAA plans to select one of these operational concepts.

NWS Plans to Respond to FAA’s Requirements, but Proposed Transition Time Frames May Be Overly Ambitious

NWS plans to respond to FAA by the May 2008 deadline, but FAA’s estimated time frames for providing the revised services may be overly ambitious. NWS plans to submit its proposals for the three operational concepts—the existing configuration located at the 21 en route centers, remote services provided by a reduced number of regional facilities, and remote services provided by a single centralized facility. FAA directed NWS to assume a transition time of 90 days for the existing configuration, 180 days for regionalized remote services, and 1 year for a single facility. To respond, NWS management created a team to analyze the requirements, gain clarification on the requirements from FAA, and develop the response. The NWS aviation services branch chief reported that the agency is on track to respond by May 2008.

However, FAA’s proposed time frames for transitioning to a new operational concept may be overly ambitious because of the activities that NWS should perform before any transition. In accordance with best practices and the opinion of the National Transportation Safety Board, NWS intends to validate that the organization can provide these new requirements—through a prototype or similar demonstration—before transitioning to a new approach. Leading organizations validate requirements to determine what impact the intended operational environment will have on the ability to satisfy the stakeholders’ needs, expectations, constraints, and interfaces. As part of this validation, organizations explore the adequacy and completeness of requirements by developing prototypes or simulations and by obtaining feedback about them from relevant stakeholders. Given the importance of accurate and timely weather information in air traffic control, it will be important for NWS to conduct a thorough evaluation before it transitions to a new operational concept in order to ensure that there are no impacts on the continuity of air traffic operations and no degradation of weather service.

In addition, NWS has agreed to negotiate with its employees’ union, the National Weather Service Employees Organization, whenever organizational changes could affect working conditions—unless the union has sufficient predecisional involvement. NWS’s employees’ union has a representative on the team that is responding to FAA’s requirements, so that later negotiations may be unnecessary. However, it is too soon to determine whether negotiations will be needed and how long they will take.

NWS’s aviation services branch chief agreed that FAA’s transition time lines would be challenging. This official estimated that it could take over 2 years to transition to a new operational concept. To address this disconnect between NWS’s capabilities and FAA’s expectations, NWS plans to propose more feasible time frames in its response to FAA. FAA officials reported that the agency will be open to NWS’s proposal.

In summary, even though center weather service units have been in operation for over two decades, neither FAA nor NWS has implemented performance measures and metrics, regularly evaluated weather service unit performance, or provided feedback to improve these aviation weather products and services. Until the agencies establish a system of performance tracking and oversight, NWS will not be able to demonstrate the quality or value of its services, and FAA will not be able to ensure the value of the services it funds. To address these shortfalls, FAA has defined more precise requirements that include performance measures and evaluation methods, and NWS is working to respond to these requirements. In its response, NWS is expected to prepare plans for three alternative operational concepts for fulfilling these requirements. FAA will choose one of the concepts. However, FAA’s proposed time frames for transitioning to a new operational concept may be overly ambitious. Given the potential for major changes to services and structure, NWS may require additional time to properly validate the requirements, plan for any necessary operational transitions, and ensure that aviation weather services are not degraded by any potential changes.

Moving forward, it is important that FAA and NWS work together to ensure that NWS’s aviation weather services address requirements and that effective performance measures and evaluation methods are established. This collaboration will help both agencies ensure the quality and consistency of these services, and ensure that FAA has the information it needs to effectively manage air traffic.

Mr. Chairman, this concludes my statement. I would be happy to answer any questions that you or members of the subcommittee may have at this time.

If you have any questions on matters discussed in this testimony, please contact me at (202) 512-9286 or by e-mail at pownerd@gao.gov. Other key contributors to this testimony include Colleen Phillips (Assistant Director), Kate Agatone, Amos Tevelow, and Jessica Waselkow.

Appendix I: Objectives, Scope, and Methodology

The objectives of this statement were to summarize selected sections of our report being released today, including National Weather Service (NWS) and Federal Aviation Administration (FAA) efforts to ensure the consistency and quality of aviation weather services at en route centers, and our recommendations to improve these services. In addition, we were asked to provide an update on FAA’s recent efforts to develop aviation weather requirements and performance measures, and NWS’s plans for responding to these requirements.

To summarize selected sections of our report, we relied on the work supporting that report. That report contains a detailed overview of our scope and methodology.

In addition, to provide an update on FAA’s recent efforts to establish requirements, we reviewed the new interagency agreement signed by both agencies in late December 2007; FAA’s requirements sent to NWS on December 19, 2007; and the accompanying quality assurance plan. We also interviewed NWS’s aviation services branch chief to clarify which of these requirements were new and which were revisions of current requirements.

To determine NWS’s plans for responding to these requirements, we reviewed the new interagency agreement, FAA’s requirements, and the accompanying quality assurance plan. We also interviewed the aviation services branch chief, who is serving in an oversight role for NWS’s response to FAA. We compared NWS’s tentative next steps with best practices for validating requirements from the Capability Maturity Model® Integration for Development.

We performed our work on the report and testimony from May 2007 to February 2008. All of the work on which this testimony is based was performed in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.