Weather Forecasting: National Weather Service's Operations Prototype Needs More Rigorous Planning
Released: 2007-07-09T00:00:00

Letter

Background

The mission of NWS—an agency within Commerce’s NOAA—is to provide weather, water, and climate forecasts and warnings for the United States, its territories, and its adjacent waters and oceans to protect life and property and to enhance the national economy. NWS is the official source of aviation- and marine-related weather forecasts and warnings as well as warnings about life-threatening weather situations.

NWS Office Structure: An Overview

The coordinated activities of weather facilities throughout the United States allow NWS to deliver a broad spectrum of climate, weather, water, and space weather services. These facilities include weather forecast offices, river forecast centers, national centers, and aviation center weather service units. The functions of these facilities are described in the following text: 122 weather forecast offices are responsible for providing a wide variety of weather, water, and climate services for their local county warning areas, including advisories, warnings, and forecasts. See figure 1 for a map of the locations of these offices. 13 river forecast centers provide river, stream, and reservoir information to a wide variety of government and commercial users as well as to local weather forecast offices for use in flood forecasts and warnings. 9 national centers constitute the National Centers for Environmental Prediction, which provide nationwide computer model output and manual forecast information to all NWS field offices and to a wide variety of government and commercial users. These centers include the Environmental Modeling Center, Storm Prediction Center, Tropical Prediction Center/National Hurricane Center, Climate Prediction Center, Aviation Weather Center, and Space Environment Center, among others. 21 aviation center weather service units, which are colocated with key Federal Aviation Administration (FAA) air traffic control centers across the nation, provide meteorological support to air traffic controllers.

NWS Relies on Key Systems and Technologies to Fulfill Its Mission

To fulfill its mission, NWS relies on a national infrastructure of systems and technologies that gather and process data from the land, sea, and air. NWS collects data from many sources, including ground-based Automated Surface Observing Systems, the Next Generation Weather Radars, and operational environmental satellites. These data are integrated by advanced data processing workstations—called an Advanced Weather Interactive Processing System (AWIPS)—and used by meteorologists to issue local forecasts and warnings. Also, these data are fed into sophisticated computer models running on high-speed supercomputers, which are then used to help develop forecasts and warnings. Figure 2 depicts the integration of the various systems and technologies and is followed by a description of each.

The Automated Surface Observing System is a system of sensors, computers, display units, and communications equipment that automates the ground-based observation and dissemination of weather information nationwide. This system collects data on temperature, dew point, visibility, wind direction and speed, pressure, cloud height and amount, and types and amounts of precipitation. There are currently 1,001 units deployed across the United States, with NWS, FAA, and the Department of Defense operating 312, 571, and 118 units, respectively.

Next Generation Weather Radar

The Next Generation Weather Radar is a Doppler radar system that detects, tracks, and determines the intensity of storms and other areas of precipitation; determines wind velocities in and around detected storm events; and generates data and imagery to help forecasters distinguish hazards, such as severe thunderstorms and tornadoes. The radar system also provides information about heavy precipitation that leads to warnings about flash floods and heavy snow. A network of these radars is made up of 158 operational radar systems and 8 nonoperational systems that are used for training and testing. Of these, NWS, the United States Air Force, and FAA sponsor 120, 26, and 12 radars, respectively.

Operational Environmental Satellites

Although NWS does not own or operate satellites, geostationary and polar- orbiting environmental satellites are key sources of data for its operations. These satellite systems continuously collect environmental data about Earth’s atmosphere, surface, and cloud cover and the electromagnetic environment. These data are used by meteorologists to develop weather forecasts and other services, and are critical to the early and reliable prediction of severe storms, such as tornadoes and hurricanes.

Advanced Weather Interactive Processing System

AWIPS is a data processing workstation that integrates and displays hydrometeorological data at NWS weather forecast offices, river forecast centers, and national centers. This system integrates data from a variety of sources (including the ground systems, radars, and satellites previously listed) to produce rich graphical displays to aid forecaster analysis and decision making. AWIPS includes a graphical forecast editing tool that allows forecasters to display and manipulate detailed graphical depictions of expected weather, and to use these depictions to generate text and graphical forecasts. AWIPS is used to disseminate weather information to the national centers; weather offices; the media; and other federal, state, and local government agencies. Figure 3 shows a standard AWIPS workstation.

Numerical weather models are advanced software programs that assimilate data from satellites and ground-based observing systems and provide short- and long-term weather pattern predictions. Meteorologists typically use a combination of models and their own experience to develop local forecasts and warnings. In addition, numerical weather models are a critical source for forecasting weather up to 2 weeks in advance and forecasting long-term climate changes.

Supercomputers

NWS leases high-performance supercomputers to execute numerical calculations supporting weather prediction and climate modeling. In 2002, NWS awarded a $227 million contract to lease high-performance supercomputers to run its environmental models from 2002 through September 2011. Included in this contract are an operational supercomputer used to run numerical weather models, an identical backup supercomputer located at a different site, and a research and development supercomputer on which researchers can test new analyses and models.

Weather Forecast Office Operations: An Overview

NWS delivers a broad spectrum of climate, weather, water, and space weather services through the coordinated activities of the weather forecast offices; river forecast centers; and national centers, which comprise the National Centers for Environmental Prediction, aviation center weather service units, and other support offices around the country. Each weather forecast office currently has a fixed geographic area for which it provides local warnings and forecasts. These offices are staffed by meteorologists who produce a suite of weather products, including short-term and 3- to 5-day extended forecasts as well as aviation and marine weather products, and monitor for severe weather events. Predictive weather services (outlooks, forecasts, watches, warnings, advisories, and other supporting information) are produced through a process that incorporates observational data, numerical model output and statistical guidance, and human expertise. These services are delivered in many forms, including textual, digital, and graphical formats. Local offices and regional and national centers coordinate as needed in the preparation of these services.

When a severe weather event occurs, forecast office managers often ask their staffs to work overtime so that there are enough personnel available to do both the normal forecasting work and the watches and warnings required by the severe event. If a weather forecast office is unable to provide forecast and warning functions due to an interruption in communications or power, an adjacent office will temporarily assume those duties. For example, during Hurricane Katrina, several weather forecast offices in Louisiana, Mississippi, and Alabama were damaged and inoperable. These offices’ responsibilities were transferred to backup sites in Texas, Florida, Alabama, and Tennessee. During this transfer of operations, called service backup, the backup office gains access to the original office’s system data and calls in extra personnel to handle the workload of both offices.

According to NWS officials, due to software limitations in AWIPS, switching to a backup service arrangement can take 60 to 90 minutes, and, when invoked, the backup office must assume all products and services for the defunct office. In addition, the current backup arrangement can cause duplicate work. For example, one office could produce a forecast for its own geographic area, and then would need to produce a separate forecast for its backup partner.

NWS Explored Options for a New Concept of Operations and Proposed One Option for Further Study

Over the past few years, NWS has evaluated options for an alternate concept of operations. According to agency officials, in 2005, NWS’s board of executive advisors (called the Corporate Board) noted that a constrained budget, high labor costs, difficulty in training and developing its employees, and a lack of flexibility in how the agency was operating were making it more difficult for the agency to continue to perform its mission. In August 2005, the board chartered a working group to evaluate the roles, responsibilities, and functions of weather offices nationwide and to make a proposal for a more efficient concept of operations. The group was given a set of guiding principles, including that the proposed concept should (1) be cost-effective, (2) ensure that there would be no degradation of service, (3) ensure that weather services nationwide were equitable, and (4) not reduce the number of forecast offices nationwide.

The group gathered input from various agency stakeholders and other partners within NOAA and considered multiple alternatives. They dismissed all but one of the alternative concepts because they were not consistent with the guiding principles. In its December 2005 proposal, the working group proposed the remaining concept, in which several offices with the same type of weather and warning responsibilities, climate, and customers would be grouped together to share responsibilities, particularly when there is a high-impact weather event affecting one of the offices. NWS called this a “clustered peer concept of operations.”

Under the clustered peer concept of operations, offices would share the workload associated with routine services, such as 7-day forecasts. During a high-impact weather event—such as a severe storm, flood, or wildfire— the offices would redistribute the workload to allow the impacted office to focus solely on the event, while the other offices in the cluster would pick up the impacted office’s routine services. In addition, offices in a cluster could occasionally allocate the work so that certain offices would take on forecasting responsibilities while other offices focused on other tasks, such as training, customer outreach, and research. In this manner, peer offices could help supplement staffing needs, and the workload across multiple offices could be more efficiently balanced.

Some of these benefits were demonstrated in recent years by a pair of weather forecasting offices. The Detroit and Grand Rapids offices established a technology workaround to allow them to more efficiently switch to backup operations between the two offices. This technology improvement allows them to switch to backup mode instantaneously, rather than taking 60 to 90 minutes using current AWIPS technology. To date, the two offices have used this backup capability to share workload during high-impact weather events; during system upgrades; and while conducting officewide training, research, and public outreach.

NWS Developed a Prototype Plan for the Alternative Concept of Operations

In February 2006, NWS’s Corporate Board chartered a team to develop a prototype plan for the clustered peer concept of operations. Between February and November 2006, the team established a preliminary concept for what the prototype would entail, conducted a laboratory exercise to determine AWIPS’s ability to support the prototype, and—on the basis of the findings from the laboratory exercise—developed a prototype plan.

The team’s preliminary concept was for the prototype to consist of four clusters ranging in size from four to eight weather forecast offices. However, in September 2006, a laboratory exercise conducted between NOAA’s Global Systems Division in Boulder and NWS’s Central Region Headquarters in Kansas City showed that AWIPS was not capable of supporting clusters of more than four offices. The laboratory results showed that with minor modifications, AWIPS was capable of allowing a more efficient and timely switch of operations between two weather forecast offices for backup purposes, and that, in some cases, functions could be shared among four offices. Officials in charge of the laboratory exercise estimated that larger clusters would not be possible until the next version of AWIPS is deployed—a milestone currently planned for 2010. The laboratory exercise also identified several problems with system performance when weather forecast offices work as a cluster of two or more offices. Specifically, each weather forecast office has its own set of customized tools supporting each office’s graphical forecast editor. The laboratory exercise found that running these tools when sharing functions among multiple offices caused AWIPS to slow down or crash.

On the basis of the findings from the laboratory exercise, the prototype team developed a plan to demonstrate opportunities for improved efficiency. The goal of the prototype was to validate that a group of weather forecast offices, operating as one unit, could be more efficient than each office operating on its own. The plan entailed sharing forecasting responsibilities between 2, and subsequently among 4, weather forecast offices using a single workstation in each office. The prototype was to be a hands-on demonstration conducted at 20 weather forecast offices, clustered in 4 different areas of the country: the Pacific Northwest, Great Plains, Great Lakes, and Southeast (see fig. 4). These areas were selected because they present different forecasting challenges, involve all of the regions in the contiguous United States, and engage a variety of field offices. In November 2006, the Corporate Board approved moving forward with the prototype plan.

NWS Initiated a 2-Year Prototype and Then Suspended It

Following the Corporate Board’s decision to proceed, NWS initiated the first of three phases of its prototype, which it planned to continue over the next 2 years. The first phase was expected to take place between November 2006 and June 2007 and was to include planning activities. As of March 2007, NWS had established a program manager and initiated efforts to plan for the prototype, including documenting how the prototype would be executed and developing training for employees who would participate in the prototype.

During the second phase, which NWS expected to take place between August 2007 and February 2008, the agency planned to have 20 weather offices share forecasting responsibilities in 2-office pairs. During the third phase, which NWS expected to take place between April 2008 and April 2009, the agency planned to have the same 20 weather offices share forecasting responsibilities in 4-office clusters. Each phase was to be preceded by both a laboratory exercise to test the new expanded functions and a board decision on whether to move ahead, based on the results of the laboratory exercise. NWS planned to conclude prototype operations and issue a report on the prototype by July 2009. By August 2009, NWS intended to make a decision on whether to implement the new concept of operations on a national basis. NWS’s schedule for the prototype is provided in figure 5. A more detailed description of the phases of the prototype is provided in appendix II.

These efforts were halted when the prototype was suspended. On March 23, 2007, Commerce’s Under Secretary for Oceans and Atmosphere directed NWS’s director to place all activities associated with NWS’s concept of operations on hold pending a review by the Deputy Under Secretary. NWS’s director subsequently issued a memorandum noting that the prototype was being suspended because of concerns about the agency’s approach expressed by some Members of Congress and by NOAA leadership, the private sector, the agency’s workforce, and the agency’s emergency management partners. The director noted that the prototype would be put on hold until a reevaluation could be undertaken.

NWS Has Not Yet Sufficiently Justified Its Prototype

Before the prototype was suspended, NWS developed and approved a 2-year plan for its prototype without conducting an analysis of the relative costs and benefits of the initiative. Leading organizations have developed best practices for justifying prototyping efforts—including analyzing the expected costs and quantified benefits of alternatives before beginning prototyping activities—and identifying plans, schedules, and risks. Furthermore, federal guidance notes that in conducting a cost-benefit assessment, tangible and intangible benefits and costs should be identified, assessed, and reported.

The prototype plan approved by the Corporate Board in November 2006 identifies the schedules, risks, and expected benefits of the prototype. The prototype was estimated to cost $9.3 million, and the Corporate Board’s financial investment review committee approved spending $1.5 million on the prototype in the first half of fiscal year 2007. As of March 2007, NWS had spent approximately $724,000 on the prototype (see table 1). Moreover, the prototype plan identifies qualitative benefits of the new concept of operations—including increased efficiency and time for other activities (such as outreach and research), the ability to switch to a backup site more quickly when a weather forecast office is disabled due to loss of power or communications, and an improved ability to focus on high-impact events.

However, NWS did not conduct a cost-benefit analysis to justify its plans for the prototype or the $1.9 million that has already been approved for the prototype.

By not conducting a cost-benefit analysis before beginning the prototype, NWS lacked assurance that its approach was cost-effective. If NWS decides to proceed with the prototype without conducting a cost-benefit analysis, the agency will have little basis to ensure that its investment is sound or that the options it is undertaking provide the greatest return on investment. Specifically, a cost-benefit analysis could help ascertain which aspects of the prototype promise the most benefits and enable NWS to adjust its plans accordingly. For example, a cost-benefit assessment might show that more benefits could be gained for the least cost by moving to a 2-office cluster, rather than a 4-office cluster.

NWS Lacks a Rigorous Evaluation Plan for Assessing Its Prototype

In its prototype plan, NWS identified goals and selected measures to evaluate during its prototype activities, but it did not establish a rigorous evaluation plan. Specifically, the agency had not defined a comprehensive set of measures, how it planned to compare prototype results with baseline performance, or how its measures supported the goals of the prototype.

According to best practices in leading organizations, well-developed evaluation plans increase the likelihood that evaluations will yield sound results, thereby supporting effective policy decisions. Key features of a well-developed evaluation plan include well-defined, clear, and measurable objectives or goals; quantitative measures for determining program performance as compared with baseline performance; and a linkage between the measures and the program goals. Such plans are essential to ensuring, among other things, that programs are executed properly and that funding requests are developed appropriately. In July 2006, we reported that NWS had not yet established plans, time lines, and measures for evaluating its prototype, and we recommended that NWS do so before beginning the prototype.

NWS’s prototype plan identifies evaluation goals and measures, and notes that NWS plans to contract with an independent entity to perform the evaluation. The prototype goals were to (1) validate that a group of weather forecast offices, operating as one unit, could be more effective and efficient than each office on its own; (2) increase focus during high- impact events through more efficient resource allocation; (3) gain efficiencies through the use of resource allocation during benign weather that can be applied to training, outreach, and professional development; and (4) use some of the efficiencies gained through the prototype to further the NOAA mission. Quantitative and qualitative measures that were to be used in evaluating the prototype included the number of times that work was transferred to another forecast office to focus on high-impact events, the number of additional hours made available during benign weather for nonforecast work, and customer perceptions of service quality and timeliness. The agency stated that system performance and hardware/software functionality would have been evaluated using staff feedback, performance monitoring software, and usage logs, while customer perceptions would have been evaluated using customer satisfaction surveys.

However, NWS did not establish an evaluation plan that identifies a full set of applicable measures, what baseline performance its prototype would be compared with, and how its measures would support the goals of the prototype. Specifically, while the agency listed various items that it planned to measure in its prototype plan, it did not include quantitative measures of product quality, timeliness, the impact on the efficiency of the office that is receiving the increased workload, or plans to assess the sensitivity of its results to weather-related situations. In addition, the agency did not establish the baseline performance against which it would compare its prototype, so it was not clear how prototype results would be assessed. Furthermore, the measures that NWS identified did not directly support the goals. For example, the number of times functions are transferred to another site does not demonstrate increased efficiency. For this measure to support the desired goal of increased efficiency, the measure would need to be supplemented with an assessment of how the office that transferred its workload used the newly available time and how the receiving site handled its increased workload.

If NWS were to proceed with its prototype without a rigorous evaluation plan, the agency would run an increased risk that it could not sufficiently measure the impact of planned changes on its performance, and would subsequently make decisions affecting the nation’s weather service on the basis of incomplete and flawed data.

NWS Involved Internal Stakeholders in Planning the Prototype, but Did Not Involve External Stakeholders or Establish a Plan for Stakeholder Involvement

NWS involved internal stakeholders in planning its prototype, including having selected stakeholders review plans and provide feedback on work products and laboratory activities. However, NWS did not involve external stakeholders or establish a plan for stakeholder involvement throughout the prototype. Leading organizations routinely involve relevant stakeholders when considering operational or process changes. In addition, NWS has agreed to negotiate with its employees’ union, the National Weather Service Employees Organization, whenever organizational changes could affect working conditions. Moreover, best practices call for developing a plan for stakeholder involvement that includes identifying relevant stakeholders, the roles and responsibilities of the relevant stakeholders, and a schedule for stakeholder involvement throughout the project.

In planning its prototype, NWS involved internal stakeholders, who were identified by the NWS Director of Strategic Plans as forecasters, other field and headquarters staff, and union representatives. For example, when deciding which offices to group into clusters, meteorologists-in-charge were asked for their input on cluster criteria and preferences. Furthermore, one meteorologist from each of eight potential prototype weather forecast offices participated in the laboratory test. Their feedback on the new concept of operations was acquired through questionnaires, electronic logs, and debriefings at the end of each week. In addition, the employees’ union provided input before the decision was made to begin the prototype. Both the initial working group and the prototype team included union representatives. Also, the union president participated in Corporate Board meetings that included discussions of the clustered peer prototype, and a union representative observed the final week of the laboratory test.

NWS also intended to work with external stakeholders (including emergency managers and other public safety officials, the media and other users of NWS information services, and the general public) after the prototype was complete to assess the results. Specifically, NWS planned to identify customer perceptions of service as one basis on which to evaluate the prototype. Furthermore, in an effort to make its plans available to employees and customers, NWS developed a public Web site that contains the plans and reports for the prototype.

However, NWS did not request input from external stakeholders, including emergency managers, during the past 20 months that it was planning and initiating the prototype. NWS officials reported that they did not involve external stakeholders because changes in the concept of operations during the prototype should not be noticeable to outside stakeholders; that is, external stakeholders should not notice any difference in service. Although this goal has merit, external stakeholders need to be more involved in the prototype for it to gain acceptance and succeed. Specifically, external stakeholders need to understand what is being tested, have input into the tests, be alert for differences under the prototype, and have formal mechanisms for sharing their observations. In addition, NWS did not have a comprehensive plan for involving stakeholders in its prototyping activities. Specifically, it did not identify key external stakeholders or stakeholder representatives, determine the responsibilities of internal or external stakeholders, or develop a time line for involving stakeholders and addressing their comments.

If NWS were to proceed with its prototype without a plan for involving internal and external stakeholders, it would not be able to ensure that employee and customer interests are anticipated, identified, and addressed.

NWS Planned to Ensure That the Prototype Would Not Degrade Its Service

NWS stated that it would ensure that there would be no degradation of service during or after its prototype, and that it had plans in place to mitigate the risk of any service degradation. Specifically, the agency conducted and planned to continue to conduct laboratory exercises to assess the impact of the prototype before moving into each phase. In addition, NWS planned to run the prototype software on a single, isolated workstation in each office, while maintaining the existing AWIPS software on all other workstations. Established procedures would allow offices to return to the baseline software and operational practices should problems have developed. Furthermore, NWS planned to monitor the performance of all activities and the quality and timeliness of products.

Looking forward, an NWS official stated that they also would ensure that there would be no degradation of service if the Corporate Board decided to implement this alternative concept of operations on a national basis after the prototype was completed. Agency officials explained that the agency would use its standard procedures for testing and validating systems and software to ensure that there would be no degradation of service before moving them into operations.

Conclusions

Although there is merit in NWS’s goal of improving its operational efficiency and allowing faster backup service during severe weather, the agency lacked the rigorous planning it needed to justify, evaluate, and obtain stakeholder involvement in its planned—and now suspended— prototype of a new concept of operations. Specifically, the agency developed a prototype plan that identified activities, gates, and decisions, but it did not perform a cost-benefit assessment to justify the prototype. Also, NWS identified goals and selected measures for evaluating its prototype, but it did not establish an evaluation plan that clearly identifies a comprehensive set of measures or how these measures would be compared with baseline performance. Furthermore, the agency involved internal stakeholders in planning for its prototype, but it did not seek input from external stakeholders or establish a plan for stakeholder involvement throughout the prototype. These matters will be important to consider should NWS decide to proceed with the prototype. Without undertaking a more rigorous approach, NWS would run an increased risk that its prototype will not be a sound investment, and that any resulting decisions could be based on flawed data and analysis. Taking this more rigorous approach to the prototype should not preclude NWS from considering options for improving the ability to switch to a backup site when a weather forecast office becomes disabled due to loss of power or communications.

Recommendations for Executive Action

We are making four recommendations to the Secretary of Commerce. If NWS decides to proceed with its prototype, we recommend the Secretary direct the Assistant Administrator for the National Weather Service to develop a cost-benefit analysis for the clustered peer approach before implementing the prototype in any weather forecasting office; develop an evaluation plan that includes a comprehensive set of measures that are linked to prototype goals and identifies the baseline performance that the prototype will be compared with before implementing the prototype in any weather forecasting office; and develop a plan for internal and external stakeholder involvement, which includes a list of relevant stakeholders, roles and responsibilities for these stakeholders, and a schedule for when stakeholders should be involved before implementing the prototype in any weather forecasting office.

In addition, we recommend that the Secretary direct the Assistant Administrator for the National Weather Service to evaluate moving forward with technology upgrades to the AWIPS processing system to allow weather forecast offices to switch to backup service more quickly during high-impact weather situations; these upgrades could be handled as an initiative that is separate and distinct from broader changes in the agency’s concept of operations.

Agency Comments and Our Evaluation

We received written comments on a draft of this report from Commerce (see app. III). In the department’s response, the Deputy Secretary of Commerce agreed with our recommendations to conduct a thorough cost- benefit analysis, develop a rigorous evaluation plan with fully defined performance measures, and involve internal and external stakeholders. The response stated that our findings were included in the Under Secretary’s recent review of the concept of operations prototype. On the basis of that review, the NOAA Administrator directed NWS to cease all activities associated with the concept of operations prototype, and to undertake a comprehensive and analytical review to determine new and revised requirements for NWS products and services. This review is expected to determine what, if any, changes are required. Should NWS decide to proceed with a new concept of operations, we believe that our recommendations to justify, evaluate, and involve stakeholders in any such changes would still be relevant and useful.

In addition, Commerce agreed with our recommendation to evaluate moving forward with technology upgrades to the AWIPS processing system to allow weather forecast offices to switch to backup service more quickly during high-impact weather situations. The department stated that NWS has established a proposal for improving its service backup capability, expects to approve requirements for the backup capability by September 2007, and plans to deploy this capability by September 2008.

As we agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution of it until 30 days from the report date. At that time, we will send copies of this report to interested congressional committees, the Secretary of Commerce, the Director of the Office of Management and Budget, and other interested parties. In addition, this report will be available at no charge on our Web site at www.gao.gov.

If you have any questions about this report, please contact me at (202) 512-9286 or by e-mail at pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) determine the current status of and plans for the prototype, (2) evaluate whether the justification for the prototype is sufficient, (3) determine whether the National Weather Service’s (NWS) plans to evaluate the prototype are adequate, (4) evaluate whether NWS is sufficiently involving stakeholders in its prototype plans, and (5) determine how NWS plans to ensure that there will be no degradation of service during and after the prototype.

To determine the current status of and plans for the prototype, we reviewed agency documentation on the planned prototype’s goals, major components, and milestones, such as the Corporate Board’s meeting minutes, the prototype plan, and the preprototype laboratory report. We also interviewed NWS officials to obtain clarifications on agency plans.

To evaluate whether the justification for the prototype is sufficient, we reviewed program documents such as the prototype plan and other NWS planning documents, presentations, and communications about the goals and costs of the prototype. We compared these planning documents with best practices for justifying process-improvement proposals by the Carnegie Mellon University Software Engineering Institute’s Capability Maturity Model®Integration for Development and federal guidance on conducting cost-benefit analyses. We also interviewed agency officials and employees who were involved in the planning of the prototype and laboratory exercise to determine the potential costs, benefits, and risks of the prototype.

To determine whether NWS plans to evaluate the prototype are adequate, we reviewed planning documents, such as the prototype plan and NWS customer satisfaction scores, and compared them with best practices for evaluating prototypes from the Capability Maturity Model®Integration for Development and GAO’s guidance for developing and assessing evaluation plans. In addition, we interviewed NWS officials who were responsible for planning the prototype activities.

To evaluate whether NWS is sufficiently involving stakeholders in its prototype plans, we reviewed NWS’s plans, presentations, and other communications about the prototype, including an external Web site that presented these documents. We compared NWS’s plans for involving stakeholders with best practices from the Capability Maturity Model® Integration for Development. We spoke with members of stakeholder groups to determine their knowledge of the prototype and their interaction with NWS. In addition, we interviewed NWS officials and the president of the National Weather Service Employees Organization to determine the level of involvement of the employee union in planning for the prototype.

To determine how NWS plans to ensure that there will be no degradation of service during and after the prototype, we reviewed planning documents—including the prototype plan, presentations, and communications regarding the prototype—and interviewed NWS officials.

We performed our work at NWS headquarters in the Washington, D.C., metropolitan area; at NWS offices in Tampa, Florida, Detroit, Michigan, and Kansas City, Missouri; and at a National Oceanic and Atmospheric Administration (NOAA) office in Boulder, Colorado. We visited the Tampa weather forecast office because the president of the employee union and the vice president of the International Association of Emergency Managers are located in the Tampa area. The Detroit weather forecast office was included because of its prior experiences in working in a paired backup with expanded domains. We visited the NWS Central Region Headquarters in Kansas City because of its participation in the first laboratory exercise and because it is where the prototype team chair worked. The NOAA Global Systems Division office in Boulder was chosen because of its participation in the first laboratory exercise. We performed our work from October 2006 to April 2007 in accordance with generally accepted government auditing standards.

Appendix II: NWS Prototype Plans, as of March 2007

Prior to the suspension of the prototype by the Director of the National Weather Service, the prototype team developed a plan for a three-phased prototype of the clustered peer concept of operations. As described in the plan, phase 1 would include planning activities, such as training; phase 2 was meant to establish clustered peer operations in 10 pairs of weather forecast offices across the country; and phase 3 would include offices grouped into 4-office clusters. NWS planned to conclude prototype operations and to issue a report on the prototype by July 2009. NWS was to decide at that time if it would proceed to implement the clustered peer concept of operations on a national basis. Details about each of the planned prototype phases are included in the following text.

Phase 1, the planning phase, began in November 2006 with the Corporate Board’s approval of the prototype plan. NWS established a program manager to oversee the prototype. In addition, cluster management teams representing each cluster were to meet to decide on the rules of engagement, common tools, and common forecast methodologies. At the end of this phase, a second laboratory exercise was planned to validate that software upgrades and hardware additions to support the prototype were operationally ready. The second laboratory exercise was scheduled for June 2007 and was to involve only weather forecast offices that are currently service-backup pairs within the original four geographic cluster areas. Upon completion of this exercise, the Corporate Board was to decide on whether to initiate Phase 2 of the prototype.

Phase 2 was planned to begin in August 2007. During this phase, NWS planned to conduct training, develop guidance for how and when offices would take over activities for other offices (called playbooks), define the cluster methodology, and have 10 pairs of offices operate as clustered peers. Also during this phase, NWS planned to develop the software upgrades needed for expanded operations in the next phase. Near the end of this phase, in February 2008, NWS planned to conduct a third laboratory exercise to test the software necessary for moving to clusters of 4 offices. After completion of this exercise, the Corporate Board was to decide on whether to initiate Phase 3 of the prototype.

Phase 3 was planned to begin in April 2008 and was to determine if clusters of 4 offices can work as a multioffice team sharing forecasting responsibilities. NWS planned to complete a preliminary prototype report in July 2009 and present it to the Corporate Board. At that time, the board would have decided whether to implement the clustered peer concept of operations across the country.

Appendix III: Comments from the Department of Commerce

Appendix IV: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the contact person named above, Colleen Phillips, Assistant Director; Kate Agatone; Neil Doherty; Amos Tevelow; and Jessica Waselkow made key contributions to this report.