Aviation Weather: Agencies Need to Improve Performance Measurement and Fully Address Key Challenges
Published: 2010-09-09T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The National Weather Service's (NWS) weather products are a vital component of the Federal Aviation Administration's (FAA) air traffic control system. In addition to providing aviation weather products developed at its own facilities, NWS also provides on-site staff at each of FAA's en route centers--the facilities that control high-altitude flight outside the airport tower and terminal areas. NWS's on-site staff is called a center weather service unit. For several years, NWS and FAA have been exploring options for improving the aviation weather services provided at en route centers. GAO agreed to (1) determine the status of the agencies' efforts to restructure aviation weather services, (2) assess the agencies' progress in establishing performance baselines in order to measure the effect of any changes, and (3) evaluate plans to address key challenges. To do so, GAO evaluated agency progress and plans and compared agency efforts with leading practices.
section_title: What GAO Found
paragraphs: After developing and shelving four proposals for restructuring the center weather service units over the last 5 years, in July 2010, senior NWS and FAA officials agreed to continue the current center weather service units at each of the 21 en route centers through September 2011 and to take immediate steps to improve aviation weather services by (1) having the service units provide forecasts for 10 key FAA terminal radar facilities and (2) having nearby weather forecast offices support FAA's en route centers when the service units are closed for the night. In addition, the agencies agreed to establish a joint team to baseline current capabilities and develop firm requirements for aviation weather services supporting air traffic flow management. While this agreement is important, the details have not been fully defined. Thus, it is not yet clear what will happen to the 21 service units after September 2011, when the immediate improvements in services will be in place, whether there are any costs associated with these steps, and who will pay for them. Until the two agencies further define their plans, the risk remains that the agencies will misjudge their responsibilities and not fulfill their agreements. FAA and NWS have made progress in identifying performance measures for the weather service units located at FAA en route centers, and NWS is beginning to track its service units' performance. However, NWS has not yet tracked, established baselines for, and reported to FAA on all applicable performance measures. Specifically, of seven possible performance measures, NWS is tracking performance for three of the measures and partially tracking a fourth measure. Of these four measures, the agency has established a sound baseline and reported on two of these measures and has made partial progress on two others. The agency is not tracking performance, documenting baselines, or reporting on three of the measures because it has not yet determined how to track them. Without an understanding of the current level of performance of the identified measures, the agencies will be limited in their ability to evaluate what progress has been made. In addition, until NWS regularly reports on its performance, the agencies lack the information they need to determine what is working well and what needs to be improved. In September 2009, GAO identified three challenges in modifying NWS's aviation weather services provided at FAA's en route centers: achieving interagency collaboration, defining requirements, and aligning changes with the Next Generation Air Transportation System (NextGen)--a long-term initiative to increase the efficiency of the national airspace system. The agencies have not yet fully addressed these challenges. Specifically, while senior agency officials recently agreed on how to proceed, work remains to be done to refine requirements, develop and execute an implementation plan, and to ensure that improvements are aligned with the long-term vision for NextGen. Until these fundamental challenges are addressed, the agencies are unlikely to achieve significant improvements in the aviation weather services provided at en route centers.
section_title: What GAO Recommends
paragraphs: GAO recommends that the Departments of Commerce and Transportation define their agreements, refine performance management processes, and address key challenges. In commenting on a draft of this report, Commerce agreed with GAO's recommendations and identified plans to address them; Transportation agreed to consider the recommendations.

section_title: Letter
paragraphs: The National Weather Service (NWS) plays a significant role in providing  weather services to the aviation community. NWS’s weather products and  data are vital components of the Federal Aviation Administration’s (FAA)  air traffic control system, providing weather information to local, regional,  and national air traffic management, navigation, and surveillance systems.  NWS aviation weather products include forecasts and warnings of  meteorological conditions that could affect air traffic, including  thunderstorms, air turbulence, and icing. In addition to providing aviation  weather products that are developed at its own facilities, NWS also  provides staff on site at each of FAA’s en route centers—the facilities that  control high-altitude flight outside the airport tower and terminal areas.  This group of NWS meteorologists—called a center weather service unit— provides air traffic staff with forecasts, advisories, and periodic weather  briefings on regional conditions.
Over the last few years, FAA and NWS have been exploring options for  enhancing the efficiency of the aviation weather services provided at en  route centers. In September 2005, FAA asked NWS to restructure its  services to be more efficient. Since then, at FAA’s request, NWS has  submitted five different proposals for restructuring its services. Because of  your interest in possible changes to NWS’s aviation weather services, we  agreed to (1) determine the status of the agencies’ efforts to restructure  aviation weather services and products, (2) assess the agencies’ progress  in establishing performance baselines in order to measure the effect of any  changes, and (3) evaluate plans to address key challenges.
To address our objectives, we analyzed agency documentation, including  FAA’s requirements for aviation weather services and NWS’s proposals.  We evaluated the agencies’ progress in establishing a baseline of  performance and compared NWS’s performance measurement practices  with government and industry guidance and best practices. To evaluate  plans to address challenges, we compared agency efforts with leading  practices in industry and government on interagency collaboration and  system development. We also interviewed officials of both agencies.
We conducted our work at NOAA and FAA facilities in the Washington,  D.C., metropolitan area. We conducted this performance audit from  October 2009 to September 2010, in accordance with generally accepted  government auditing standards. Those standards require that we plan and  perform the audit to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.  Additional details on our objectives, scope, and methodology are provided  in appendix I.
subsections: section_title: Background
paragraphs: FAA is responsible for ensuring safe, orderly, and efficient air travel in and  around the United States. NWS supports FAA by providing aviation-related  forecasts and warnings at air traffic facilities across the country. Among  other support and services, NWS provides four meteorologists at each of  FAA’s 21 en route centers to provide on-site aviation weather services.  This arrangement is defined and funded under an interagency agreement.
subsections: section_title: FAA’s Mission and Organizational Structure
paragraphs: In performing its primary mission to ensure safe air travel, FAA reported  that air traffic in the national airspace system exceeded 43 million flights  and 745 million passengers in 2008. In addition, at any one time, as many  as 7,000 aircraft—both civilian and military—could be aloft over the  United States. In 2004, FAA’s Air Traffic Organization was formed to,  among other responsibilities, improve the provision of air traffic services.  More than 34,000 employees within FAA’s Air Traffic Organization support  the operations that help move aircraft through the national airspace  system. The agency’s ability to fulfill its mission depends on the adequacy  and reliability of its air traffic control systems, as well as weather  forecasts made available by NWS and automated systems. These resources  reside at, or are associated with, several types of facilities: air traffic  control towers, terminal radar approach control facilities, air route traffic  control centers (en route centers), and the Air Traffic Control System  Command Center. The number and functions of these facilities are as  follows:    510 air traffic control towers manage and control the airspace within  about 5 miles of an airport. They control departures and landings, as well  as ground operations on airport taxiways and runways.   163 terminal radar approach control facilities provide air traffic control  services for airspace within approximately 40 miles of an airport and  generally up to 10,000 feet above the airport, where en route centers’  control begins. Terminal controllers establish and maintain the sequence  and separation of aircraft.    21 en route centers control planes over the United States—in transit and  during approaches to some airports. Each center handles a different  region of airspace. En route centers operate the computer suite that  processes radar surveillance and flight planning data, reformats the data  for presentation purposes, and sends it to display equipment used by  controllers to track aircraft. The centers control the switching of voice  communications between aircraft and the center, as well as between the  center and other air traffic control facilities. Four of these en route centers  also control air traffic over the oceans.
The Air Traffic Control System Command Center manages the flow of air  traffic within the United States. This facility regulates air traffic when  weather, equipment, runway closures, or other conditions place stress on  the national airspace system. In these instances, traffic management  specialists at the command center take action to modify traffic demands in  order to keep traffic within system capacity.
See figure 1 for a visual summary of the facilities that control and manage  air traffic over the United States.
section_title: NWS’s Mission and Organizational Structure
paragraphs: The mission of NWS—an agency within the Department of Commerce’s  National Oceanic and Atmospheric Administration (NOAA)—is to provide  weather, water, and climate forecasts and warnings for the United States,  its territories, and its adjacent waters and oceans to protect life and  property and to enhance the national economy. In addition, NWS is the  official source of aviation- and marine-related weather forecasts and  warnings, as well as warnings about life-threatening weather situations.
The coordinated activities of weather facilities throughout the United  States allow NWS to deliver a broad spectrum of climate, weather, water,  and space weather services in support of its mission. These facilities  include 122 weather forecast offices located across the country that  provide a wide variety of weather, water, and climate services for their  local county warning areas, including advisories, warnings, and forecasts;  9 national prediction centers that provide nationwide computer modeling  to all NWS field offices; and 21 center weather service units that are  located at FAA en route centers across the nation and provide  meteorological support to air traffic controllers.
section_title: NWS Provides Aviation Weather Services to FAA
paragraphs: As an official source of aviation weather forecasts and warnings, several  NWS facilities provide aviation weather products and services to FAA and  the aviation sector. These facilities include the Aviation Weather Center,  weather forecast offices located across the country, and 21 center weather  service units located at FAA en route centers across the country.
subsections: section_title: Aviation Weather Center
paragraphs: The Aviation Weather Center located in Kansas City, Missouri, issues  warnings, forecasts, and analyses of hazardous weather for aviation.  Staffed by 55 personnel, the center develops warnings of hazardous  weather for aircraft in flight and forecasts of weather conditions for the  next 2 days that could affect both domestic and international aviation.
The center also produces a Collaborative Convective Forecast Product, a  graphical representation of expected thunderstorms or related conditions  at 2, 4, and 6 hours. This is used by FAA to manage aviation traffic flow  across the country. The Aviation Weather Center’s key products are  described in table 1.
NWS’s 122 weather forecast offices issue terminal area forecasts for  approximately 632 locations every 6 hours or when conditions change,  consisting of the expected weather conditions significant to a given airport  or terminal area, and are primarily used by commercial and general  aviation pilots. The terminal area forecasts are updated every 3 hours for  35 key airports and every 2 hours for the airports in New York, Atlanta,  and Chicago.
section_title: Center Weather Service Units
paragraphs: NWS’s center weather service units are located at each of FAA’s 21 en  route centers and operate 16 hours a day, 7 days a week (see fig. 2). Each  center weather service unit usually consists of three meteorologists and a  meteorologist-in-charge who provide strategic advice and aviation weather  forecasts to FAA traffic management personnel. Governed by an  interagency agreement, FAA currently reimburses NWS approximately $13  million annually for this support.
section_title: Center Weather Service Units: An Overview of Systems and Operations
paragraphs: The meteorologists at the center weather service units use a variety of  systems to gather and analyze information compiled from NWS and FAA  weather sensors. Key systems used to compile weather information  include FAA’s Weather and Radar Processor, FAA’s Integrated Terminal  Weather System, FAA’s Corridor Integrated Weather System, and a remote  display of NWS’s Advanced Weather Interactive Processing System.  Meteorologists at several center weather service units also use NWS’s  National Centers—Advanced Weather Interactive Processing System.  Table 2 provides a description of key systems.
NWS meteorologists at the en route centers provide several products and  services to the FAA staff, including meteorological impact statements,  center weather advisories, periodic briefings, and on-demand  consultations. These products and services are described in table 3. In  addition, center weather service unit meteorologists receive and  disseminate pilot reports, provide input every 2 hours to the Aviation  Weather Center’s creation of the Collaborative Convective Forecast  Product, train FAA personnel on how to interpret weather information,  and provide weather briefings to nearby terminal radar approach control  facilities and air traffic control towers.
section_title: FAA Seeks to Improve Aviation Weather Services Provided at En Route Centers
paragraphs: In recent years, FAA has undertaken multiple initiatives to assess and  improve the performance of the center weather service units. Studies  conducted in 2003 and 2006 highlighted concerns with the lack of  standardization of products and services at NWS’s center weather service  units. To address these concerns, the agency sponsored studies that  determined that weather data could be provided remotely using current  technologies, and that private sector vendors could provide these services.  In 2005, the agency requested that NWS restructure its aviation weather  services by consolidating its center weather service units to a smaller  number of sites, reducing personnel costs, and providing products and  services 24 hours a day, 7 days a week. NWS subsequently submitted a  proposal for restructuring its services, but FAA declined the proposal  citing the need to refine its requirements.
In December 2007, FAA issued revised requirements and asked NWS to  respond with proposals defining the technical and cost implications of  three operational concepts. The three concepts involved (1) on-site  services provided within the existing configuration of offices located at  the 21 en route centers, (2) remote services provided by a reduced number  of regional facilities, and (3) remote services provided by a single  centralized facility. NWS responded with three proposals, but FAA  rejected these proposals in September 2008, noting that while elements of  each proposal had merit, the proposed costs were too high. FAA requested  that NWS revise its proposal to bring costs down while stating a  preference to move toward a single center weather service unit with a  backup site.
As a separate initiative, NWS began a series of improvements in order to  address FAA’s key concerns. Specifically, in April 2008, the agency  initiated a program to improve the consistency of the center weather  service units’ products and services. This program involved standardizing  the technology, collaboration, and training for all 21 center weather  service units and conducting site visits to evaluate and provide feedback  to each unit. NWS reported that it completed these efforts in 2009. A  summary of FAA’s key concerns and NWS’s efforts to address them is  included in appendix II.
After two requests for deadline extensions on a new proposal, NWS  provided FAA with an updated proposal in June 2009 based on the two-site  approach FAA had requested in September 2008. FAA responded to NWS’s  proposal by requesting more information and stated that the agencies  would work together to resolve issues. From September through  November 2009, the agencies completed a series of meetings to address  issues from the proposal and agreed that NWS would resubmit its proposal  in December 2009 to consolidate the service units. In December 2009, FAA  revised its requirements to reflect the agencies’ efforts aimed at improving  center weather service operations. However, NWS did not submit its  proposal in December 2009 to consolidate the center weather service  units. According to NWS officials, they decided not to submit the proposal  because (1) the NWS labor union and others raised concerns about  consolidating offices, (2) NWS could implement technical improvements  more quickly under the current organizational structure, and (3) the  agency wanted to focus its efforts and resources on future weather system  development rather than restructuring existing operations. Table 4  provides a chronology of the agencies’ assessment and improvement  efforts.
section_title: Prior GAO Reports Identified Concerns with Center Weather Service Units; Recommended Steps to Improve Quality Assurance
paragraphs: In January 2008, we reported on concerns about inconsistencies in  products and quality among center weather service units. We noted that  while both NWS and FAA have responsibilities for assuring and controlling  the quality of aviation weather observations, neither agency monitored the  accuracy and quality of the aviation weather products provided at center  weather service units, performed annual evaluations of aviation weather  services provided at en route centers, and provided feedback to the center  weather service units. We recommended they do so. The Department of  Commerce agreed with our recommendations, and the Department of  Transportation stated that FAA planned to revise its requirements and that  these would establish performance measures and evaluation procedures.
In September 2009, we reported that the agencies were considering plans  to consolidate 20 of the 21 existing center weather service units to two  locations, but it was not clear whether and how the changes would be  implemented. Moreover, we reported that NWS and FAA faced challenges  in their efforts to improve the aviation weather structure, including  achieving interagency collaboration, defining FAA’s requirements, and  aligning any changes with the Next Generation Air Transportation System.  We also identified three challenges the agencies would face in  implementing their plans—developing a feasible schedule that includes  adequate time for stakeholder involvement, undertaking a comprehensive  demonstration to ensure no services are degraded, and effectively  reconfiguring the infrastructure and technologies. We recommended that  the agencies address these challenges, and NOAA and the Department of  Transportation agreed with our recommendations.
section_title: NWS and FAA Officials Agreed to Improve Services, but This Agreement Is Not Fully Defined
paragraphs: After developing and shelving four different proposals for restructuring  the center weather service units over the last 5 years, NWS and FAA have  reached agreement on how to improve aviation weather services. In March  2010, NWS proposed maintaining the current 21 center weather service  units collocated at en route centers, increasing staffing at the Aviation  Weather Center in order to provide remote services during the service  units’ off-hours, and developing a new collaborative weather product.  NWS estimated that these improvements would cost FAA about $3 million  per year. This is in addition to the annual cost of maintaining the existing  21 centers. NWS also estimated that it would be able to implement the  proposal within 21 months. FAA responded that it was not prepared to  accept the proposal because of the increased costs.
Subsequently, in July 2010, FAA and NWS reached an agreement on the  steps the two agencies would take to improve aviation weather services.  Specifically, FAA proposed and NWS agreed to continue the current center  weather service units at each of the 21 en route centers through  September 2011 and to take immediate steps to improve aviation weather  services by (1) having the service units provide forecasts at 10 key FAA  terminal radar approach control facilities and (2) providing around-the- clock coverage at all of the en route centers by having the local weather  forecast office support the en route centers when the center weather  service units are closed for the night—a practice that currently is used at  selected en route centers. In addition, the agencies agreed to establish a  joint team to baseline current capabilities and develop firm requirements  for NWS products and services supporting FAA’s air traffic flow  management out through 2015. The agencies expect that the joint team  will establish an implementation plan by November 2010 and then begin to  implement it.
However, the agencies’ documentation of this agreement does not address  the future locations of the center weather service units, or provide details  and a schedule for the proposed improvements to services. As a result, it is  not clear what will happen to the 21 service units after September 2011,  when the immediate improvements in services will be in place, whether  there are any costs associated with these steps, whether the benefits  outweigh the costs, and who will pay for them. Until this agreement is  further defined in writing and formalized between the two agencies, the  risks remain that the agencies will misjudge their responsibilities and not  fulfill their agreements.
section_title: NWS Has Made Progress in Measuring Center Weather Service Unit Performance but Has Not Yet Established Performance Baselines or Reported on Key Measures
paragraphs: According to best practices in the federal government and in industry,  organizations should measure performance in order to evaluate the  success or failure of their activities and programs. Performance  measurement involves identifying performance goals and measures,  establishing performance baselines by tracking performance over time,  identifying targets for improving performance, and measuring progress  against those targets. In January 2008, we recommended that NWS and  FAA develop performance measures and track metrics for the products  and services provided by center weather service units and that they  provide feedback to the center weather service units so that they could  improve their performance. Further, in September 2009, we  recommended that the agencies approve their draft performance measures  and establish performance baselines so that they could understand the  effects of any changes from restructuring aviation weather services.
Over the past year, NWS has made progress in identifying performance  measures, tracking performance on selected measures, and reporting on  the selected measures; however, the agency is not yet tracking or  reporting on all applicable performance measures. In December 2008, FAA  provided NWS five performance measures of center weather service unit  performance. Under the current interagency agreement, NWS is required  to track and report to FAA on these measures. In addition, in its last two  proposals, NWS proposed additional measures, two of which could be  tracked under the current organizational structure and using current  products. We previously recommended that NWS immediately identify  the current level of performance of the proposed measures that could be  identified under the current organizational structure, so that they will have  a performance baseline to compare to should they decide to implement  operational changes. The agency agreed with this recommendation. Table  5 describes the performance areas applicable to the current center  weather service unit structure.
NWS has started tracking performance for three of the seven measures  and is partially tracking a fourth. Specifically, NWS has tracked data on  each center weather service unit’s (1) participation in the Collaborative  Convective Forecast Product, (2) organizational service provision, and (3)  customer satisfaction. Further, it has partially tracked data on format  consistency, by collecting data on one of two required products. However,  the agency has not tracked data on the other measures for a number of  reasons. For example, the agency did not track the format consistency for  the second of the two required products because, until recently, the  briefing has not had a consistent format. Also, the agency is not tracking  training completion because it has not yet determined what standardized  training will be provided. For the forecast accuracy measure, agency  officials stated that they do not currently have the means to track this  measure, but that they are developing a tool to do so.
Of the measures it is tracking, NWS has established baselines and reported  its results on two measures and has partially done so for two other  measures. Specifically, NWS has established baselines on each center  weather service unit’s participation in the Collaborative Convective  Forecast Product and its organizational service provision. In addition,  NWS has partially established a baseline on the format consistency  measure in that it has historical data for one of the two required products.
However, because it has not tracked the format consistency of the second  product, NWS has not established a complete baseline for that measure.  Further, while NWS has calculated customer satisfaction scores from its  2009 site evaluations, it does not yet have a reliable baseline because it has  not yet matured its approach to documenting this measure. Specifically,  NWS changed its approach during its 2010 site evaluations, which will  make it harder to compare scores from year to year. Moreover, the agency  mixed positive and negative findings to come up with its rating scores for  some sites, thereby rendering the 2009 scores at selected sites ineffective  at measuring a site’s performance. Figure 3 identifies NWS’s efforts to  track data, develop baselines, and report on the performance  measurement areas.
It is important for NWS and FAA to track performance in the identified  measures in order to understand the value currently provided and to  assess the impact of any changes they make to operations. Reporting also  helps improve performance. For example, after reporting on its  performance in product participation and organizational service provision  for 2009, NWS noted significant improvements in 2010.
Until the agencies track and develop a performance baseline for all  applicable measures, they will be limited in their ability to evaluate  progress that has been made and whether or not they are achieving their  goals. In addition, until NWS regularly reports on its performance, the  agencies lack the information they need to determine what is working well  and what needs to be improved. Moreover, as the agencies refine their  approach to performance measurement, it will be important to revisit and  refine the performance measures to ensure an appropriate mix of process-  and outcome-oriented measures. For example, NWS could consider  measuring the number of aircraft incidents attributed to inaccurate  aviation weather forecasts or the number of weather-related delays as a  percentage of all delays.
section_title: NWS and FAA Have Not Fully Addressed Key Challenges
paragraphs: In September 2009, we identified three challenges that FAA and NWS  faced in modifying the current aviation weather structure: (1) achieving  interagency collaboration, (2) defining requirements, (3) aligning changes  with the Next Generation Air Transportation System (NextGen)—a long- term initiative to increase the efficiency of the national airspace system.  The agencies have taken initial steps to collaborate, refine requirements,  and look for ways to align their plans with NextGen, but they have not yet  fully addressed the challenges. Until these fundamental challenges are  addressed, the agencies are unlikely to achieve significant improvements  in the aviation weather services provided at en route centers.
subsections: section_title: Achieving Interagency Collaboration
paragraphs: We have previously reported on key practices that can help enhance and  sustain interagency collaboration. The practices generally consist of two  or more agencies defining a common outcome, establishing joint strategies  to achieve the outcome, agreeing upon agency roles and responsibilities,  establishing compatible policies and procedures to operate across agency  boundaries, and developing mechanisms to monitor, evaluate, and report  the results of collaborative efforts.
In September 2009, we reported that NWS and FAA had not defined a  common outcome for modifying the aviation weather services provided at  en route centers, established joint strategies, or agreed upon their  respective responsibilities. We recommended that the agencies complete  these activities. NOAA and the Department of Transportation agreed with  our recommendation.
Since September 2009, NWS and FAA have made progress in defining a  common outcome, but have not yet established joint strategies to achieve  the outcome or agreed upon agency responsibilities. Specifically, in July  2010, the two agencies defined a common outcome when they reached an  agreement to continue the current center weather service unit  configuration at each of the 21 en route centers and to take immediate  steps to improve aviation weather services. The two agencies also plan to  form a team that will develop an implementation plan by November 2010.  However, the agreement does not provide the details needed to establish  joint strategies and only provides general agency responsibilities. Until the  agencies establish joint strategies and agree on respective agency  responsibilities, it may prove difficult to move forward in efforts to  improve aviation weather services.
section_title: Defining Requirements
paragraphs: According to best practices of leading organizations, requirements  describe the functionality needed to meet user needs and perform as  intended in the operational environment. A disciplined process for  developing and managing requirements can help reduce the risks  associated with developing or acquiring a system or product.
In September 2009, we reported that FAA’s requirements were unstable  and recommended that the agencies establish and finalize requirements  for aviation weather services at en route centers. NOAA and the  Department of Transportation agreed with our recommendation.
FAA updated its requirements in December 2009 based on the work that  the two agencies did in the fall of 2009. However, these changes were  nullified by the more recent decision to continue with 21 center weather  service units. In July 2010, the two agencies agreed to establish a joint  team to develop firm requirements for NWS products and services  supporting FAA’s air traffic flow management out to 2015, including those  provided by the center weather service units. While this is an important  step, significant work remains to be done to revise these requirements.  Until the requirements are in place, the agencies may find it difficult to  move forward in efforts to improve aviation weather services.
section_title: Alignment with the Next Generation Air Transportation System
paragraphs: In September 2009, we reported that neither FAA nor NWS had ensured  that their restructuring plans fit with the national vision for NextGen—a  long-term initiative to transition FAA from the current radar-based system  to an aircraft-centered, satellite-based system. We recommended that the  agencies ensure that any proposed organizational changes are aligned by  seeking a review by the Joint Planning and Development Office, the  office responsible for planning and coordinating NextGen. NOAA and the  Department of Transportation agreed with our recommendation.
Among other agreements in July 2010, the two agencies plan to work  together to develop requirements and an implementation plan that extends  through 2015—the NextGen Midterm Operating Capability date—by  November 2010. However, because this plan has not been developed or  approved, it is not clear that future actions will be aligned with NextGen.  As NWS and FAA discuss the current proposal and plan improvements to  aviation weather services, it will be important for the agencies to continue  to ensure alignment with the long-term goals of NextGen.
section_title: Conclusions
paragraphs: After many years of proposals and counterproposals for improving the  center weather service units, NWS and FAA recently agreed to continue  the current center weather service unit configuration at each of the 21 en  route centers through September 2011 and to take immediate steps to  improve aviation weather services. However, many questions remain  about what will happen, when, and at what cost. Given the long history of  unresolved issues between FAA and NWS regarding the center weather  service units, it is more important than ever that the two agencies be  extremely clear on what their commitments entail.
An important component of any effort to improve operations is a solid  understanding of current performance. While NWS has made progress in  measuring the performance of the center weather service units, is not  adequately documenting performance baselines or reporting on several of  its performance measures. Further, the agency has begun efforts to  measure customer satisfaction, but the process is immature, and the  results are unreliable. Specifically, NWS has changed its approach to the  annual evaluations making it difficult to compare performance from year  to year, and its scoring process mixes positive and negative findings for  several sites. As a result, the scores may not accurately reflect each  center’s performance. Until NWS has a solid understanding of the current  level of performance, it will be limited in its ability to evaluate what  progress has been made and whether or not it is achieving its goals.
As the agencies move forward with plans to make aviation weather  services more efficient, they continue to face challenges, including a  record of false starts on interagency collaboration, unstable requirements,  and a lack of assurance that operational changes will align with the future  vision of NextGen. Until these challenges are fully addressed, the agencies  will likely find it difficult to make meaningful changes in aviation weather  services.
section_title: Recommendations for Executive Action
paragraphs: To improve the aviation weather products and services provided at FAA’s  en route centers, we are making three recommendations to the Secretaries  of Commerce and Transportation. Specifically, we recommend that the  Secretaries direct the NWS and FAA Administrators to    define, document, and sign the agencies’ recent agreements on (1) the  locations of the center weather service units, (2) immediate improvements  in aviation weather services and operating hours, and (3) the development  of an implementation plan for improvements through 2015;   ensure that NWS regularly tracks progress, documents performance  baselines, and reports on its format consistency, forecast accuracy, and  training performance measures; and    ensure that NWS develops a reliable customer satisfaction baseline by  refining the questions used during annual evaluations, so that comparable  information is collected from year to year, and revising the scoring process  to ensure that scores accurately reflect each center’s performance.
In addition, we are reiterating our prior recommendations to the two  agencies to address key challenges in achieving interagency collaboration,  defining requirements, and aligning any organizational changes with plans  for NextGen.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from the Secretary  of Commerce, who transmitted NOAA’s comments (see app. III). In its  comments, NOAA stated that our report is generally representative of  challenges facing NWS and FAA in the execution of aviation weather  services provided by the center weather service units. The agency agreed  with our recommendations and identified plans to implement selected  parts of the recommendations. Specifically, NOAA reiterated its plan to  form a joint NWS/FAA team to determine weather requirements for traffic  flow management and to implement products and services through the  year 2015. NOAA stated that this team’s results will serve as additional  documentation of the agreements. In addition, NOAA reported that it plans  to begin measuring format consistency in September 2010 and forecast  accuracy in December 2010.
NOAA also noted that it was using 2009 site evaluations as the basis for its  scoring and that the 2009 results would serve as a baseline for comparison  to 2010 and subsequent results. However, as we discuss in the report, our  analysis of the 2009 site evaluations and scoring process found that the  results were not reliable because the process for collecting information on  customer service was inconsistent, and the scores did not always  accurately reflect the centers’ performance. As a result, the 2009 scores  are not useful as a baseline or as a feedback tool. Moving forward, as  NOAA analyzes the results from its ongoing 2010 site evaluations, it will be  important to ensure that the scores accurately reflect each center’s  performance. Further, in future years, it will be important to ensure that  comparable information is collected from year to year so that a reliable  performance baseline can be established.
The Department of Transportation’s Director of Audit Relations provided  comments on a draft of this report via e-mail. In those comments, he noted  that the department agreed to consider our recommendations.
Both departments also provided technical comments that we incorporated  as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  appropriate congressional committees, the Secretary of Commerce, the  Secretary of Transportation, the Director of the Office of Management and  Budget, and other interested parties. The report also will be available at no  charge on the GAO Web site at http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-9286 or by e-mail at pownerd@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) determine the status of the agencies’ efforts to  restructure aviation weather services and products, (2) assess the  agencies’ progress in establishing performance baselines in order to  measure the effect of any changes, and (3) evaluate plans to address key  challenges.
To determine the status of the agencies’ efforts to restructure aviation  weather services and products, we analyzed Federal Aviation  Administration (FAA) and National Weather Service (NWS)  documentation, including FAA’s requirements for center weather service  units, the interagency agreement between FAA and NWS, and NWS’s  proposals to meet FAA needs for center weather service units. We also  interviewed officials from both agencies to discuss their plans and status  in reaching a decision on proposed changes.
To assess the agencies’ progress in establishing performance baselines, we  identified the agencies’ previous efforts to establish baselines and  evaluated the extent to which they have made progress in doing so. We  analyzed NWS’s approach to measuring center weather service unit  performance and compared its performance measurement practices with  guidance and best practices in performance management identified by  government and industry. Specifically, we assessed the agencies’ actions  taken to identify performance measures, track them, establish baselines of  performance, and report on those baselines. We also assessed the  reliability of the performance data that NWS reported. Specifically, for the  customer satisfaction measurements, we analyzed supporting data and  calculated customer satisfaction scores using NWS’s guidance for  developing scores. We then compared the scores we calculated with  NWS’s scores. In instances where our scores did not match NWS’s, we  interviewed agency officials in order to determine why NWS’s scores did  not match our own, focusing on four sites with the largest number of  findings. We found that the agency’s customer satisfaction data was not  reliable. For the other reported measures, we evaluated supporting data  and interviewed responsible agency officials to determine the agency’s  processes for validating the data. We found that the data reported for  these performance measures was sufficient to meet our reporting  purposes.
To evaluate plans to address key challenges identified in our prior report,  we reviewed agency documents including FAA requirements, an NWS  proposal, plans for the Next Generation Air Transportation System  (NextGen), and FAA’s response to NWS’s proposal. We compared agency  efforts with leading practices in industry and government on interagency  collaboration and system development. In addition, we interviewed the  FAA contracting officer’s technical representative for the center weather  service units to discuss the challenges the agency would have in  implementing NWS’s proposal, as well as the agency’s plans to ensure  requirements were stabilized. We also interviewed NWS officials to discuss  their plans for aligning their system development initiatives with NextGen.  We also interviewed the co-chair of the weather working group of the  Joint Planning and Development Office to determine whether the office  had reviewed NWS’s proposal and if the office had concerns about the  proposal’s impact on NextGen.
We conducted our work at National Oceanic and Atmospheric  Administration (NOAA) and FAA facilities in the Washington, D.C.,  metropolitan area. We conducted this performance audit from October  2009 to September 2010, in accordance with generally accepted  government auditing standards. Those standards require that we plan and  perform the audit to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.
section_title: Appendix II: Key FAA Concerns and NWS Steps to Address Them
paragraphs: Table 6 lists concerns that FAA identified in a series of studies between  2003 and 2006, as well as the steps that NWS has taken to address these  concerns.
section_title: Appendix III: Comments from the Department of Commerce
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Colleen Phillips, Assistant  Director; Neil Doherty; Rebecca Eyler; Joshua Leiling; and Jessica  Waselkow made key contributions to this report.