Weather Forecasting: National Weather Service's Operations Prototype Needs More Rigorous Planning
Published: 2007-06-08T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Using advanced systems and trained specialists located in 122 weather forecast offices throughout the country, the National Weather Service (NWS) provides storm and flood warnings and weather forecasts to protect life and property and to enhance the national economy. To improve the efficiency of its operations, in November 2006, NWS approved an effort to develop a prototype of an alternative way of operating. Under this prototype, weather forecasting offices would share selected responsibilities. GAO (1) determined the status of and plans for the prototype, (2) evaluated whether the prototype's justification was sufficient, (3) determined whether NWS's plans to evaluate the prototype are adequate, (4) evaluated whether NWS is sufficiently involving stakeholders in its prototype plans, and (5) determined how NWS plans to ensure that there will be no degradation of service during and after the prototype. To do so, GAO analyzed agency documentation and interviewed program officials and stakeholders.
section_title: What GAO Found
paragraphs: NWS's prototype is currently on hold pending a reevaluation of the agency's approach. The agency had just begun the first phase of its three-phased prototype to demonstrate a new concept of operations over a 2-year period when, in late March 2007, the Department of Commerce's Under Secretary suspended the prototype because of concerns about the agency's approach. In the first phase, NWS established a program manager and began planning for the next two phases. During the remaining phases, NWS planned to have 20 weather forecasting offices share responsibilities in 2-office pairs and then in 4-office clusters. NWS then planned to decide whether to implement the new concept of operations on a national basis. The justification for the prototype was not sufficient. Before the prototype was suspended, the agency had approved moving forward with its prototype without conducting a cost-benefit analysis. NWS estimated that the prototype would cost approximately $9.3 million and would offer qualitative benefits, such as increased efficiency and an improved ability to focus on severe weather events, but did not quantify benefits or the expected return on its investment. If NWS were to proceed with the prototype without a cost-benefit analysis, it would lack assurance that its approach would be a cost-effective investment for the agency. NWS identified goals and selected measures to evaluate during its prototype activities, but it did not establish a rigorous evaluation plan. Specifically, NWS did not define a full set of needed measures, how it planned to compare prototype results with baseline performance, or how its selected measures supported the prototype goals. If NWS were to proceed without a rigorous evaluation plan, the agency would run an increased risk of not sufficiently measuring the impact of changes on its performance and could make decisions affecting the nation's weather on the basis of incomplete or flawed data. Although NWS involved internal stakeholders in planning its prototype, it did not (1) involve external stakeholders or (2) establish a plan that identified key stakeholders, the stakeholders' responsibilities, and a time line for involving stakeholders and addressing their comments. If NWS were to proceed with its prototype without such a plan, it could not ensure that stakeholder interests would be identified and addressed. NWS planned to mitigate the risk of degradation during and after the prototype by conducting laboratory exercises to understand the impact of the prototype, using a dedicated workstation in each office so that the office could switch to original systems if warranted, and monitoring its systems and products during the prototype. NWS officials stated that should the agency decide to implement the prototype on a national basis, it has standard procedures for testing and validating systems and software to avoid any degradation of service.

section_title: Letter
paragraphs: The National Weather Service’s (NWS) ability to forecast the weather  affects the life and property of every American. The agency’s basic mission  is to provide storm and flood warnings and weather forecasts for the  United States, its territories, and adjacent oceans and waters, in order to  protect life and property and to enhance the national economy. NWS also  supports other federal agencies’ operations by providing aviation- and  marine-related weather forecasts and warnings. To fulfill its mission, NWS  operates 122 weather forecasting offices and other specialized weather  centers across the country. In these offices, trained meteorologists use  advanced systems to provide forecasts and warnings for local geographic  areas.
In an effort to improve its operational efficiency, in August 2005, NWS  chartered a working group to evaluate the roles, responsibilities,  functions, and supporting technology of weather forecast offices  nationwide and to make a proposal for a more efficient concept of  operations. In December 2005, the working group proposed an alternative  way of operating in which weather offices could share some weather  forecasting functions—particularly during high-intensity weather events.  After receiving this proposal, NWS chartered another team to develop  plans for a prototype of this concept to more fully evaluate this approach.
Because of your interest in weather service operations, we (1) determined  the current status of and plans for the prototype, (2) evaluated whether  the justification for the prototype was sufficient, (3) determined whether  NWS’s plans to evaluate the prototype are adequate, (4) evaluated whether  NWS is sufficiently involving stakeholders in its prototype plans, and   (5) determined how NWS plans to ensure that there will be no degradation  of service during and after the prototype.
In late March 2007, after our audit work was nearly completed, the  Department of Commerce’s Under Secretary for Oceans and Atmosphere  directed the Director of the National Weather Service to place all activities  associated with the agency’s concept of operations on hold pending a  review by the Deputy Under Secretary. NWS officials stated that the  prototype was suspended because of concerns raised about the agency’s  approach by some Members of Congress and by National Oceanic and  Atmospheric Administration (NOAA) leadership, the private sector, NWS’s  workforce, and its critical emergency management partners. Shortly after  this decision was made, we met with your staffs and agreed to continue  with plans to issue this report to offer insight and recommendations to  NWS, should it decide to proceed with the prototype.
To address our objectives, we reviewed prototype plans and presentations.  We compared program documents with best practices for justifying and  evaluating prototypes and for involving stakeholders. We also interviewed  internal and external stakeholders and agency officials. We conducted our  work at NWS headquarters in the Washington, D.C., metropolitan area;  NWS offices in Tampa, Florida, Detroit, Michigan, and Kansas City,  Missouri; and a NOAA office in Boulder, Colorado, because of the  relevance of these sites to the prototype. We performed our work from  October 2006 to April 2007, in accordance with generally accepted  government auditing standards. Additional details on our objectives,  scope, and methodology are provided in appendix I.
subsections: section_title: Background
paragraphs: The mission of NWS—an agency within Commerce’s NOAA—is to provide  weather, water, and climate forecasts and warnings for the United States,  its territories, and its adjacent waters and oceans to protect life and  property and to enhance the national economy. NWS is the official source  of aviation- and marine-related weather forecasts and warnings as well as  warnings about life-threatening weather situations.
subsections: section_title: NWS Office Structure: An Overview
paragraphs: The coordinated activities of weather facilities throughout the United  States allow NWS to deliver a broad spectrum of climate, weather, water,  and space weather services. These facilities include weather forecast  offices, river forecast centers, national centers, and aviation center  weather service units. The functions of these facilities are described in the  following text:    122 weather forecast offices are responsible for providing a wide variety of  weather, water, and climate services for their local county warning areas,  including advisories, warnings, and forecasts. See figure 1 for a map of the  locations of these offices.   13 river forecast centers provide river, stream, and reservoir information  to a wide variety of government and commercial users as well as to local  weather forecast offices for use in flood forecasts and warnings.   9 national centers constitute the National Centers for Environmental  Prediction, which provide nationwide computer model output and manual  forecast information to all NWS field offices and to a wide variety of  government and commercial users. These centers include the  Environmental Modeling Center, Storm Prediction Center, Tropical  Prediction Center/National Hurricane Center, Climate Prediction Center,  Aviation Weather Center, and Space Environment Center, among others.   21 aviation center weather service units, which are colocated with key  Federal Aviation Administration (FAA) air traffic control centers across  the nation, provide meteorological support to air traffic controllers.
section_title: NWS Relies on Key Systems and Technologies to Fulfill Its Mission
paragraphs: To fulfill its mission, NWS relies on a national infrastructure of systems  and technologies that gather and process data from the land, sea, and air.  NWS collects data from many sources, including ground-based Automated  Surface Observing Systems, the Next Generation Weather Radars, and  operational environmental satellites. These data are integrated by  advanced data processing workstations—called an Advanced Weather  Interactive Processing System (AWIPS)—and used by meteorologists to  issue local forecasts and warnings. Also, these data are fed into  sophisticated computer models running on high-speed supercomputers,  which are then used to help develop forecasts and warnings. Figure 2  depicts the integration of the various systems and technologies and is  followed by a description of each.
The Automated Surface Observing System is a system of sensors,  computers, display units, and communications equipment that automates  the ground-based observation and dissemination of weather information  nationwide. This system collects data on temperature, dew point, visibility,  wind direction and speed, pressure, cloud height and amount, and types  and amounts of precipitation. There are currently 1,001 units deployed  across the United States, with NWS, FAA, and the Department of Defense  operating 312, 571, and 118 units, respectively.
subsections: section_title: Next Generation Weather Radar
paragraphs: The Next Generation Weather Radar is a Doppler radar system that  detects, tracks, and determines the intensity of storms and other areas of  precipitation; determines wind velocities in and around detected storm  events; and generates data and imagery to help forecasters distinguish  hazards, such as severe thunderstorms and tornadoes. The radar system  also provides information about heavy precipitation that leads to warnings  about flash floods and heavy snow. A network of these radars is made up  of 158 operational radar systems and 8 nonoperational systems that are  used for training and testing. Of these, NWS, the United States Air Force,  and FAA sponsor 120, 26, and 12 radars, respectively.
section_title: Operational Environmental Satellites
paragraphs: Although NWS does not own or operate satellites, geostationary and polar- orbiting environmental satellites are key sources of data for its  operations. These satellite systems continuously collect environmental  data about Earth’s atmosphere, surface, and cloud cover and the  electromagnetic environment. These data are used by meteorologists to  develop weather forecasts and other services, and are critical to the early  and reliable prediction of severe storms, such as tornadoes and  hurricanes.
section_title: Advanced Weather Interactive Processing System
paragraphs: AWIPS is a data processing workstation that integrates and displays  hydrometeorological data at NWS weather forecast offices, river forecast  centers, and national centers. This system integrates data from a variety of  sources (including the ground systems, radars, and satellites previously  listed) to produce rich graphical displays to aid forecaster analysis and  decision making. AWIPS includes a graphical forecast editing tool that  allows forecasters to display and manipulate detailed graphical depictions  of expected weather, and to use these depictions to generate text and  graphical forecasts. AWIPS is used to disseminate weather information to  the national centers; weather offices; the media; and other federal, state,  and local government agencies. Figure 3 shows a standard AWIPS  workstation.
Numerical weather models are advanced software programs that  assimilate data from satellites and ground-based observing systems and  provide short- and long-term weather pattern predictions. Meteorologists  typically use a combination of models and their own experience to  develop local forecasts and warnings. In addition, numerical weather  models are a critical source for forecasting weather up to 2 weeks in  advance and forecasting long-term climate changes.
section_title: Supercomputers
paragraphs: NWS leases high-performance supercomputers to execute numerical  calculations supporting weather prediction and climate modeling. In 2002,  NWS awarded a $227 million contract to lease high-performance  supercomputers to run its environmental models from 2002 through  September 2011. Included in this contract are an operational  supercomputer used to run numerical weather models, an identical  backup supercomputer located at a different site, and a research and  development supercomputer on which researchers can test new analyses  and models.
section_title: Weather Forecast Office Operations: An Overview
paragraphs: NWS delivers a broad spectrum of climate, weather, water, and space  weather services through the coordinated activities of the weather  forecast offices; river forecast centers; and national centers, which  comprise the National Centers for Environmental Prediction, aviation  center weather service units, and other support offices around the  country. Each weather forecast office currently has a fixed geographic  area for which it provides local warnings and forecasts. These offices are  staffed by meteorologists who produce a suite of weather products,  including short-term and 3- to 5-day extended forecasts as well as aviation  and marine weather products, and monitor for severe weather events.  Predictive weather services (outlooks, forecasts, watches, warnings,  advisories, and other supporting information) are produced through a  process that incorporates observational data, numerical model output and  statistical guidance, and human expertise. These services are delivered in  many forms, including textual, digital, and graphical formats. Local offices  and regional and national centers coordinate as needed in the preparation  of these services.
When a severe weather event occurs, forecast office managers often ask  their staffs to work overtime so that there are enough personnel available  to do both the normal forecasting work and the watches and warnings  required by the severe event. If a weather forecast office is unable to  provide forecast and warning functions due to an interruption in  communications or power, an adjacent office will temporarily assume  those duties. For example, during Hurricane Katrina, several weather  forecast offices in Louisiana, Mississippi, and Alabama were damaged and  inoperable. These offices’ responsibilities were transferred to backup sites  in Texas, Florida, Alabama, and Tennessee. During this transfer of  operations, called service backup, the backup office gains access to the  original office’s system data and calls in extra personnel to handle the  workload of both offices.
According to NWS officials, due to software limitations in AWIPS,  switching to a backup service arrangement can take 60 to 90 minutes, and,  when invoked, the backup office must assume all products and services  for the defunct office. In addition, the current backup arrangement can  cause duplicate work. For example, one office could produce a forecast  for its own geographic area, and then would need to produce a separate  forecast for its backup partner.
section_title: NWS Explored Options for a New Concept of Operations and Proposed One Option for Further Study
paragraphs: Over the past few years, NWS has evaluated options for an alternate  concept of operations. According to agency officials, in 2005, NWS’s board  of executive advisors (called the Corporate Board) noted that a  constrained budget, high labor costs, difficulty in training and developing  its employees, and a lack of flexibility in how the agency was operating  were making it more difficult for the agency to continue to perform its  mission. In August 2005, the board chartered a working group to evaluate  the roles, responsibilities, and functions of weather offices nationwide and  to make a proposal for a more efficient concept of operations. The group  was given a set of guiding principles, including that the proposed concept  should (1) be cost-effective, (2) ensure that there would be no degradation  of service, (3) ensure that weather services nationwide were equitable, and  (4) not reduce the number of forecast offices nationwide.
The group gathered input from various agency stakeholders and other  partners within NOAA and considered multiple alternatives. They  dismissed all but one of the alternative concepts because they were not  consistent with the guiding principles. In its December 2005 proposal, the  working group proposed the remaining concept, in which several offices  with the same type of weather and warning responsibilities, climate, and  customers would be grouped together to share responsibilities,  particularly when there is a high-impact weather event affecting one of the  offices. NWS called this a “clustered peer concept of operations.”
Under the clustered peer concept of operations, offices would share the  workload associated with routine services, such as 7-day forecasts. During  a high-impact weather event—such as a severe storm, flood, or wildfire— the offices would redistribute the workload to allow the impacted office to  focus solely on the event, while the other offices in the cluster would pick  up the impacted office’s routine services. In addition, offices in a cluster  could occasionally allocate the work so that certain offices would take on  forecasting responsibilities while other offices focused on other tasks,  such as training, customer outreach, and research. In this manner, peer  offices could help supplement staffing needs, and the workload across  multiple offices could be more efficiently balanced.
Some of these benefits were demonstrated in recent years by a pair of  weather forecasting offices. The Detroit and Grand Rapids offices  established a technology workaround to allow them to more efficiently  switch to backup operations between the two offices. This technology  improvement allows them to switch to backup mode instantaneously,  rather than taking 60 to 90 minutes using current AWIPS technology. To  date, the two offices have used this backup capability to share workload  during high-impact weather events; during system upgrades; and while  conducting officewide training, research, and public outreach.
section_title: NWS Developed a Prototype Plan for the Alternative Concept of Operations
paragraphs: In February 2006, NWS’s Corporate Board chartered a team to develop a  prototype plan for the clustered peer concept of operations. Between  February and November 2006, the team established a preliminary concept  for what the prototype would entail, conducted a laboratory exercise to  determine AWIPS’s ability to support the prototype, and—on the basis of  the findings from the laboratory exercise—developed a prototype plan.
The team’s preliminary concept was for the prototype to consist of four  clusters ranging in size from four to eight weather forecast offices.  However, in September 2006, a laboratory exercise conducted between  NOAA’s Global Systems Division in Boulder and NWS’s Central Region  Headquarters in Kansas City showed that AWIPS was not capable of  supporting clusters of more than four offices. The laboratory results  showed that with minor modifications, AWIPS was capable of allowing a  more efficient and timely switch of operations between two weather  forecast offices for backup purposes, and that, in some cases, functions  could be shared among four offices. Officials in charge of the laboratory  exercise estimated that larger clusters would not be possible until the next  version of AWIPS is deployed—a milestone currently planned for 2010.  The laboratory exercise also identified several problems with system  performance when weather forecast offices work as a cluster of two or  more offices. Specifically, each weather forecast office has its own set of  customized tools supporting each office’s graphical forecast editor. The  laboratory exercise found that running these tools when sharing functions  among multiple offices caused AWIPS to slow down or crash.
On the basis of the findings from the laboratory exercise, the prototype  team developed a plan to demonstrate opportunities for improved  efficiency. The goal of the prototype was to validate that a group of  weather forecast offices, operating as one unit, could be more efficient  than each office operating on its own. The plan entailed sharing  forecasting responsibilities between 2, and subsequently among 4, weather  forecast offices using a single workstation in each office. The prototype  was to be a hands-on demonstration conducted at 20 weather forecast  offices, clustered in 4 different areas of the country: the Pacific Northwest,  Great Plains, Great Lakes, and Southeast (see fig. 4). These areas were  selected because they present different forecasting challenges, involve all  of the regions in the contiguous United States, and engage a variety of field  offices. In November 2006, the Corporate Board approved moving forward  with the prototype plan.
section_title: NWS Initiated a 2-Year Prototype and Then Suspended It
paragraphs: Following the Corporate Board’s decision to proceed, NWS initiated the  first of three phases of its prototype, which it planned to continue over the  next 2 years. The first phase was expected to take place between  November 2006 and June 2007 and was to include planning activities. As of  March 2007, NWS had established a program manager and initiated efforts  to plan for the prototype, including documenting how the prototype would  be executed and developing training for employees who would participate  in the prototype.
During the second phase, which NWS expected to take place between  August 2007 and February 2008, the agency planned to have 20 weather  offices share forecasting responsibilities in 2-office pairs. During the third  phase, which NWS expected to take place between April 2008 and April  2009, the agency planned to have the same 20 weather offices share  forecasting responsibilities in 4-office clusters. Each phase was to be  preceded by both a laboratory exercise to test the new expanded functions  and a board decision on whether to move ahead, based on the results of  the laboratory exercise. NWS planned to conclude prototype operations  and issue a report on the prototype by July 2009. By August 2009, NWS  intended to make a decision on whether to implement the new concept of  operations on a national basis. NWS’s schedule for the prototype is  provided in figure 5. A more detailed description of the phases of the  prototype is provided in appendix II.
These efforts were halted when the prototype was suspended. On March  23, 2007, Commerce’s Under Secretary for Oceans and Atmosphere  directed NWS’s director to place all activities associated with NWS’s  concept of operations on hold pending a review by the Deputy Under  Secretary. NWS’s director subsequently issued a memorandum noting that  the prototype was being suspended because of concerns about the  agency’s approach expressed by some Members of Congress and by NOAA  leadership, the private sector, the agency’s workforce, and the agency’s  emergency management partners. The director noted that the prototype  would be put on hold until a reevaluation could be undertaken.
section_title: NWS Has Not Yet Sufficiently Justified Its Prototype
paragraphs: Before the prototype was suspended, NWS developed and approved a   2-year plan for its prototype without conducting an analysis of the relative  costs and benefits of the initiative. Leading organizations have developed  best practices for justifying prototyping efforts—including analyzing the  expected costs and quantified benefits of alternatives before beginning  prototyping activities—and identifying plans, schedules, and risks.  Furthermore, federal guidance notes that in conducting a cost-benefit  assessment, tangible and intangible benefits and costs should be  identified, assessed, and reported.
The prototype plan approved by the Corporate Board in November 2006  identifies the schedules, risks, and expected benefits of the prototype. The  prototype was estimated to cost $9.3 million, and the Corporate Board’s  financial investment review committee approved spending $1.5 million on  the prototype in the first half of fiscal year 2007. As of March 2007, NWS  had spent approximately $724,000 on the prototype (see table 1).  Moreover, the prototype plan identifies qualitative benefits of the new  concept of operations—including increased efficiency and time for other  activities (such as outreach and research), the ability to switch to a  backup site more quickly when a weather forecast office is disabled due to  loss of power or communications, and an improved ability to focus on  high-impact events.
However, NWS did not conduct a cost-benefit analysis to justify its plans  for the prototype or the $1.9 million that has already been approved for the  prototype.
By not conducting a cost-benefit analysis before beginning the prototype,  NWS lacked assurance that its approach was cost-effective. If NWS  decides to proceed with the prototype without conducting a cost-benefit  analysis, the agency will have little basis to ensure that its investment is  sound or that the options it is undertaking provide the greatest return on  investment. Specifically, a cost-benefit analysis could help ascertain which  aspects of the prototype promise the most benefits and enable NWS to  adjust its plans accordingly. For example, a cost-benefit assessment might  show that more benefits could be gained for the least cost by moving to a  2-office cluster, rather than a 4-office cluster.
section_title: NWS Lacks a Rigorous Evaluation Plan for Assessing Its Prototype
paragraphs: In its prototype plan, NWS identified goals and selected measures to  evaluate during its prototype activities, but it did not establish a rigorous  evaluation plan. Specifically, the agency had not defined a comprehensive  set of measures, how it planned to compare prototype results with  baseline performance, or how its measures supported the goals of the  prototype.
According to best practices in leading organizations, well-developed  evaluation plans increase the likelihood that evaluations will yield sound  results, thereby supporting effective policy decisions. Key features of a  well-developed evaluation plan include well-defined, clear, and  measurable objectives or goals; quantitative measures for determining  program performance as compared with baseline performance; and a  linkage between the measures and the program goals. Such plans are  essential to ensuring, among other things, that programs are executed  properly and that funding requests are developed appropriately. In July  2006, we reported that NWS had not yet established plans, time lines, and  measures for evaluating its prototype, and we recommended that NWS do  so before beginning the prototype.
NWS’s prototype plan identifies evaluation goals and measures, and notes  that NWS plans to contract with an independent entity to perform the  evaluation. The prototype goals were to (1) validate that a group of  weather forecast offices, operating as one unit, could be more effective  and efficient than each office on its own; (2) increase focus during high- impact events through more efficient resource allocation; (3) gain  efficiencies through the use of resource allocation during benign weather  that can be applied to training, outreach, and professional development;  and (4) use some of the efficiencies gained through the prototype to  further the NOAA mission. Quantitative and qualitative measures that were  to be used in evaluating the prototype included the number of times that  work was transferred to another forecast office to focus on high-impact  events, the number of additional hours made available during benign  weather for nonforecast work, and customer perceptions of service quality  and timeliness. The agency stated that system performance and  hardware/software functionality would have been evaluated using staff  feedback, performance monitoring software, and usage logs, while  customer perceptions would have been evaluated using customer  satisfaction surveys.
However, NWS did not establish an evaluation plan that identifies a full set  of applicable measures, what baseline performance its prototype would be  compared with, and how its measures would support the goals of the  prototype. Specifically, while the agency listed various items that it  planned to measure in its prototype plan, it did not include quantitative  measures of product quality, timeliness, the impact on the efficiency of the  office that is receiving the increased workload, or plans to assess the  sensitivity of its results to weather-related situations. In addition, the  agency did not establish the baseline performance against which it would  compare its prototype, so it was not clear how prototype results would be  assessed. Furthermore, the measures that NWS identified did not directly  support the goals. For example, the number of times functions are  transferred to another site does not demonstrate increased efficiency. For  this measure to support the desired goal of increased efficiency, the  measure would need to be supplemented with an assessment of how the  office that transferred its workload used the newly available time and how  the receiving site handled its increased workload.
If NWS were to proceed with its prototype without a rigorous evaluation  plan, the agency would run an increased risk that it could not sufficiently  measure the impact of planned changes on its performance, and would  subsequently make decisions affecting the nation’s weather service on the  basis of incomplete and flawed data.
section_title: NWS Involved Internal Stakeholders in Planning the Prototype, but Did Not Involve External Stakeholders or Establish a Plan for Stakeholder Involvement
paragraphs: NWS involved internal stakeholders in planning its prototype, including  having selected stakeholders review plans and provide feedback on work  products and laboratory activities. However, NWS did not involve external  stakeholders or establish a plan for stakeholder involvement throughout  the prototype. Leading organizations routinely involve relevant  stakeholders when considering operational or process changes. In  addition, NWS has agreed to negotiate with its employees’ union, the  National Weather Service Employees Organization, whenever  organizational changes could affect working conditions. Moreover, best  practices call for developing a plan for stakeholder involvement that  includes identifying relevant stakeholders, the roles and responsibilities of  the relevant stakeholders, and a schedule for stakeholder involvement  throughout the project.
In planning its prototype, NWS involved internal stakeholders, who were  identified by the NWS Director of Strategic Plans as forecasters, other field  and headquarters staff, and union representatives. For example, when  deciding which offices to group into clusters, meteorologists-in-charge  were asked for their input on cluster criteria and preferences.  Furthermore, one meteorologist from each of eight potential prototype  weather forecast offices participated in the laboratory test. Their feedback  on the new concept of operations was acquired through questionnaires,  electronic logs, and debriefings at the end of each week. In addition, the  employees’ union provided input before the decision was made to begin  the prototype. Both the initial working group and the prototype team  included union representatives. Also, the union president participated in  Corporate Board meetings that included discussions of the clustered peer  prototype, and a union representative observed the final week of the  laboratory test.
NWS also intended to work with external stakeholders (including  emergency managers and other public safety officials, the media and other  users of NWS information services, and the general public) after the  prototype was complete to assess the results. Specifically, NWS planned to  identify customer perceptions of service as one basis on which to evaluate  the prototype. Furthermore, in an effort to make its plans available to  employees and customers, NWS developed a public Web site that contains  the plans and reports for the prototype.
However, NWS did not request input from external stakeholders, including  emergency managers, during the past 20 months that it was planning and  initiating the prototype. NWS officials reported that they did not involve  external stakeholders because changes in the concept of operations during  the prototype should not be noticeable to outside stakeholders; that is,  external stakeholders should not notice any difference in service.  Although this goal has merit, external stakeholders need to be more  involved in the prototype for it to gain acceptance and succeed.  Specifically, external stakeholders need to understand what is being  tested, have input into the tests, be alert for differences under the  prototype, and have formal mechanisms for sharing their observations. In  addition, NWS did not have a comprehensive plan for involving  stakeholders in its prototyping activities. Specifically, it did not identify  key external stakeholders or stakeholder representatives, determine the  responsibilities of internal or external stakeholders, or develop a time line  for involving stakeholders and addressing their comments.
If NWS were to proceed with its prototype without a plan for involving  internal and external stakeholders, it would not be able to ensure that  employee and customer interests are anticipated, identified, and  addressed.
section_title: NWS Planned to Ensure That the Prototype Would Not Degrade Its Service
paragraphs: NWS stated that it would ensure that there would be no degradation of  service during or after its prototype, and that it had plans in place to  mitigate the risk of any service degradation. Specifically, the agency  conducted and planned to continue to conduct laboratory exercises to  assess the impact of the prototype before moving into each phase. In  addition, NWS planned to run the prototype software on a single, isolated  workstation in each office, while maintaining the existing AWIPS software  on all other workstations. Established procedures would allow offices to  return to the baseline software and operational practices should problems  have developed. Furthermore, NWS planned to monitor the performance  of all activities and the quality and timeliness of products.
Looking forward, an NWS official stated that they also would ensure that  there would be no degradation of service if the Corporate Board decided  to implement this alternative concept of operations on a national basis  after the prototype was completed. Agency officials explained that the  agency would use its standard procedures for testing and validating  systems and software to ensure that there would be no degradation of  service before moving them into operations.
section_title: Conclusions
paragraphs: Although there is merit in NWS’s goal of improving its operational  efficiency and allowing faster backup service during severe weather, the  agency lacked the rigorous planning it needed to justify, evaluate, and  obtain stakeholder involvement in its planned—and now suspended— prototype of a new concept of operations. Specifically, the agency  developed a prototype plan that identified activities, gates, and decisions,  but it did not perform a cost-benefit assessment to justify the prototype.  Also, NWS identified goals and selected measures for evaluating its  prototype, but it did not establish an evaluation plan that clearly identifies  a comprehensive set of measures or how these measures would be  compared with baseline performance. Furthermore, the agency involved  internal stakeholders in planning for its prototype, but it did not seek input  from external stakeholders or establish a plan for stakeholder involvement  throughout the prototype. These matters will be important to consider  should NWS decide to proceed with the prototype. Without undertaking a  more rigorous approach, NWS would run an increased risk that its  prototype will not be a sound investment, and that any resulting decisions  could be based on flawed data and analysis. Taking this more rigorous  approach to the prototype should not preclude NWS from considering  options for improving the ability to switch to a backup site when a  weather forecast office becomes disabled due to loss of power or  communications.
section_title: Recommendations for Executive Action
paragraphs: We are making four recommendations to the Secretary of Commerce. If  NWS decides to proceed with its prototype, we recommend the Secretary  direct the Assistant Administrator for the National Weather Service to    develop a cost-benefit analysis for the clustered peer approach before  implementing the prototype in any weather forecasting office;   develop an evaluation plan that includes a comprehensive set of measures  that are linked to prototype goals and identifies the baseline performance  that the prototype will be compared with before implementing the  prototype in any weather forecasting office; and   develop a plan for internal and external stakeholder involvement, which  includes a list of relevant stakeholders, roles and responsibilities for these  stakeholders, and a schedule for when stakeholders should be involved  before implementing the prototype in any weather forecasting office.
In addition, we recommend that the Secretary direct the Assistant  Administrator for the National Weather Service to evaluate moving  forward with technology upgrades to the AWIPS processing system to  allow weather forecast offices to switch to backup service more quickly  during high-impact weather situations; these upgrades could be handled as  an initiative that is separate and distinct from broader changes in the  agency’s concept of operations.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from Commerce  (see app. III). In the department’s response, the Deputy Secretary of  Commerce agreed with our recommendations to conduct a thorough cost- benefit analysis, develop a rigorous evaluation plan with fully defined  performance measures, and involve internal and external stakeholders.  The response stated that our findings were included in the Under  Secretary’s recent review of the concept of operations prototype. On the  basis of that review, the NOAA Administrator directed NWS to cease all  activities associated with the concept of operations prototype, and to  undertake a comprehensive and analytical review to determine new and  revised requirements for NWS products and services. This review is  expected to determine what, if any, changes are required. Should NWS  decide to proceed with a new concept of operations, we believe that our  recommendations to justify, evaluate, and involve stakeholders in any  such changes would still be relevant and useful.
In addition, Commerce agreed with our recommendation to evaluate  moving forward with technology upgrades to the AWIPS processing  system to allow weather forecast offices to switch to backup service more  quickly during high-impact weather situations. The department stated that  NWS has established a proposal for improving its service backup  capability, expects to approve requirements for the backup capability by  September 2007, and plans to deploy this capability by September 2008.
As we agreed with your offices, unless you publicly announce the contents  of this report earlier, we plan no further distribution of it until 30 days  from the report date. At that time, we will send copies of this report to  interested congressional committees, the Secretary of Commerce, the  Director of the Office of Management and Budget, and other interested  parties. In addition, this report will be available at no charge on our Web  site at www.gao.gov.
If you have any questions about this report, please contact me at   (202) 512-9286 or by e-mail at pownerd@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this report. GAO staff who made major contributions to this  report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) determine the current status of and plans for  the prototype, (2) evaluate whether the justification for the prototype is  sufficient, (3) determine whether the National Weather Service’s (NWS)  plans to evaluate the prototype are adequate, (4) evaluate whether NWS is  sufficiently involving stakeholders in its prototype plans, and   (5) determine how NWS plans to ensure that there will be no degradation  of service during and after the prototype.
To determine the current status of and plans for the prototype, we  reviewed agency documentation on the planned prototype’s goals, major  components, and milestones, such as the Corporate Board’s meeting  minutes, the prototype plan, and the preprototype laboratory report. We  also interviewed NWS officials to obtain clarifications on agency plans.
To evaluate whether the justification for the prototype is sufficient, we  reviewed program documents such as the prototype plan and other NWS  planning documents, presentations, and communications about the goals  and costs of the prototype. We compared these planning documents with  best practices for justifying process-improvement proposals by the  Carnegie Mellon University Software Engineering Institute’s Capability  Maturity Model®Integration for Development and federal guidance on  conducting cost-benefit analyses. We also interviewed agency officials  and employees who were involved in the planning of the prototype and  laboratory exercise to determine the potential costs, benefits, and risks of  the prototype.
To determine whether NWS plans to evaluate the prototype are adequate,  we reviewed planning documents, such as the prototype plan and NWS  customer satisfaction scores, and compared them with best practices for   evaluating prototypes from the Capability Maturity Model®Integration  for Development and GAO’s guidance for developing and assessing  evaluation plans. In addition, we interviewed NWS officials who were  responsible for planning the prototype activities.
To evaluate whether NWS is sufficiently involving stakeholders in its  prototype plans, we reviewed NWS’s plans, presentations, and other  communications about the prototype, including an external Web site that  presented these documents. We compared NWS’s plans for involving  stakeholders with best practices from the Capability Maturity Model®  Integration for Development. We spoke with members of stakeholder  groups to determine their knowledge of the prototype and their interaction  with NWS. In addition, we interviewed NWS officials and the president of  the National Weather Service Employees Organization to determine the  level of involvement of the employee union in planning for the prototype.
To determine how NWS plans to ensure that there will be no degradation  of service during and after the prototype, we reviewed planning  documents—including the prototype plan, presentations, and  communications regarding the prototype—and interviewed NWS officials.
We performed our work at NWS headquarters in the Washington, D.C.,  metropolitan area; at NWS offices in Tampa, Florida, Detroit, Michigan,  and Kansas City, Missouri; and at a National Oceanic and Atmospheric  Administration (NOAA) office in Boulder, Colorado. We visited the Tampa  weather forecast office because the president of the employee union and  the vice president of the International Association of Emergency Managers  are located in the Tampa area. The Detroit weather forecast office was  included because of its prior experiences in working in a paired backup  with expanded domains. We visited the NWS Central Region Headquarters  in Kansas City because of its participation in the first laboratory exercise  and because it is where the prototype team chair worked. The NOAA   Global Systems Division office in Boulder was chosen because of its  participation in the first laboratory exercise. We performed our work from  October 2006 to April 2007 in accordance with generally accepted  government auditing standards.
section_title: Appendix II: NWS Prototype Plans, as of March 2007
paragraphs: Prior to the suspension of the prototype by the Director of the National  Weather Service, the prototype team developed a plan for a three-phased  prototype of the clustered peer concept of operations. As described in the  plan, phase 1 would include planning activities, such as training; phase 2  was meant to establish clustered peer operations in 10 pairs of weather  forecast offices across the country; and phase 3 would include offices  grouped into 4-office clusters. NWS planned to conclude prototype  operations and to issue a report on the prototype by July 2009. NWS was  to decide at that time if it would proceed to implement the clustered peer  concept of operations on a national basis. Details about each of the  planned prototype phases are included in the following text.
Phase 1, the planning phase, began in November 2006 with the Corporate  Board’s approval of the prototype plan. NWS established a program  manager to oversee the prototype. In addition, cluster management teams  representing each cluster were to meet to decide on the rules of  engagement, common tools, and common forecast methodologies. At the  end of this phase, a second laboratory exercise was planned to validate  that software upgrades and hardware additions to support the prototype  were operationally ready. The second laboratory exercise was scheduled  for June 2007 and was to involve only weather forecast offices that are  currently service-backup pairs within the original four geographic cluster  areas. Upon completion of this exercise, the Corporate Board was to  decide on whether to initiate Phase 2 of the prototype.
Phase 2 was planned to begin in August 2007. During this phase, NWS  planned to conduct training, develop guidance for how and when offices  would take over activities for other offices (called playbooks), define the  cluster methodology, and have 10 pairs of offices operate as clustered  peers. Also during this phase, NWS planned to develop the software  upgrades needed for expanded operations in the next phase. Near the end  of this phase, in February 2008, NWS planned to conduct a third laboratory  exercise to test the software necessary for moving to clusters of 4 offices.  After completion of this exercise, the Corporate Board was to decide on  whether to initiate Phase 3 of the prototype.
Phase 3 was planned to begin in April 2008 and was to determine if  clusters of 4 offices can work as a multioffice team sharing forecasting  responsibilities. NWS planned to complete a preliminary prototype report  in July 2009 and present it to the Corporate Board. At that time, the board  would have decided whether to implement the clustered peer concept of  operations across the country.
section_title: Appendix III: Comments from the Department of Commerce
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contact person named above, Colleen Phillips, Assistant  Director; Kate Agatone; Neil Doherty; Amos Tevelow; and Jessica  Waselkow made key contributions to this report.