United States General Accounting Office

GAO Report to the Secretary of Commerce

May 1996 WEATHER

FORECASTING

Recommendations to

Address New Weather

Processing System

Development Risks

G O A

years

1921 - 1996

GAO/AIMD-96-74

GAO United States

General Accounting Office

Washington, D.C. 20548

Accounting and Information

Management Division

B-271478 

May 13, 1996

The Honorable Mickey Kantor

The Secretary of Commerce

Dear Mr. Secretary:

On February 29, 1996, we testified on the National Weather Service’s (NWS)

Advanced Weather Interactive Processing System (AWIPS)—the

cornerstone of NWS’ $4.5 billion modernization program—before the

Subcommittee on Energy and Environment, House Committee on Science.1

This report officially transmits recommendations we made as part of our

testimony which, if implemented, will strengthen NWS’ ability to achieve a

fair return on its AWIPS investment.

First, as we also recommended in our February 1996 report2

 on AWIPS, NWS

should (1) expand ongoing AWIPS requirements review activities to include

validation that proposed capabilities are justified on the basis of mission

impact and (2) implement only those capabilities that are validated. These

recommendations are grounded in NWS’ long-standing assertion that the

modernization and its component systems will produce significant public

service improvements—namely, better forecasts at reduced costs. To

achieve these goals, NWS has specified that AWIPS must provide about 450

high-order capabilities, such as the ability to execute certain models or

display data in certain formats and colors. These high-order capabilities

are composed of about 22,000 individual system requirements.

We testified that while NWS has done a commendable job of incorporating

the views of the forecasting community in defining AWIPS’ capabilities, it

has not demonstrated that all proposed capabilities will produce expected

mission impacts. We have found that successful public and private

organizations tie decisions on information technology investments to

explicit and quantifiable mission improvements.3

 By failing to do the same,

NWS risks building either more or less into AWIPS than necessary, which,

either way, translates into lowering AWIPS’ return-on-investment.

1

Weather Forecasting: New Processing System Faces Uncertainties and Risks (GAO/T-AIMD-96-47,

February 29, 1996).

2

Weather Forecasting: NWS Has Not Demonstrated That New Processing System Will Improve Mission

Effectiveness (GAO/AIMD-96-29, February 29, 1996).

3

Executive Guide: Improving Mission Performance Through Strategic Information Management and

Technology (GAO/AIMD-94-115, May 1994).

Page 1 GAO/AIMD-96-74 Weather Forecasting Recommendations
B-271478 

Second, we testified that NWS has wisely chosen to break AWIPS software

development into increments, which is a widely accepted means of

reducing software development risks. However, we believe that NWS’

approach to implementing this strategy makes AWIPS development

unnecessarily risky because NWS chose to overlap development of these

builds, beginning development of a new build before the prior build is fully

stabilized. In doing so, it increased the chances of potential defects being

passed along module-to-module. Accordingly, we recommended that NWS

ensure that each build is fully tested and all material defects are corrected

before software development associated with the next build begins.

Third, we recommended in our testimony that NWS establish a software

quality assurance program to increase the probability of delivering

promised AWIPS capability on time and within budget. While NWS has

reported a number of software development process improvements,

including preparing a software development plan, we believe its failure to

establish a software quality assurance program to monitor NWS’ in-house

development processes and products unnecessarily puts AWIPS’ current

cost, schedule, and performance goals in jeopardy. Such a program would

provide independent assurance that software development processes and

products meet prescribed standards and that any identified deficiencies in

process or product are brought to management’s attention sooner rather

than later, thus minimizing costly system rework. Employing a software

quality assurance program is a common practice among public and private

organizations with solid reputations for producing quality software.

Finally, in our testimony, we recommended that NWS obtain an

independent assessment of the cost to develop and deploy AWIPS. We

testified that NWS currently does not have reliable cost information on

AWIPS development and deployment costs and that its current estimate of

$525 million, which has not changed for more than 1 year, omits certain

costs and is thus understated. For example:

• The $525 million estimate includes neither known deployment phase

contract cost increases nor NWS internal program management costs

resulting from NWS’ 1995 decision to extend AWIPS deployment 1 year. In

our discussions with NWS officials, contract costs associated with the

extension were estimated to be about $9 million, while NWS program

management costs for the additional year had yet to be estimated.

• The estimate omits a known cost increase in the development phase

contract of about $10 million. While NWS officials stated that this increase

Page 2 GAO/AIMD-96-74 Weather Forecasting Recommendations
B-271478 

will be offset by reductions in the deployment phase contract, the terms of

the deployment phase have yet to be renegotiated.

• The estimate omits the cost of AWIPS product improvements that were

recently identified and are planned for 1999. These improvements are

estimated to cost about $2 million.

• The $525 million estimate, when originally derived, included about

$49 million for future development and deployment phase contract

modifications—a $34 million decrease from the amount NWS included in its

1992 project cost estimate. Since 1994, however, NWS officials told us that

more than $42 million of the $49 million in reserve has been allocated to

known contract costs, leaving relatively meager reserves for inevitable

modifications. Omitting adequate reserves at this stage in AWIPS’

development cycle is inconsistent with prudent software engineering

practices. According to software engineering experts, the size of a

system’s software at this stage in its development life cycle could grow by

as much as 50 percent.

As we testified, NWS has an opportunity with AWIPS to substantially improve

its weather forecasting capability. If the recommendations we made in our

testimony and in this report are implemented, NWS can position itself to

take full advantage of this opportunity.

Agency Comments

and Our Evaluation

In its written comments on a draft of this report, the National Oceanic and

Atmospheric Administration (NOAA) generally disagreed with our

conclusions and recommendations. A summary of NOAA’s specific

comments and our evaluation of them follows. We have incorporated

other NOAA comments in the report where appropriate.

NOAA disagreed with our first recommendation to (1) expand ongoing AWIPS

validation activities to include justifying proposed capabilities on the basis

of mission impact and (2) implement only mission-justified capabilities.

According to NOAA, completed and ongoing requirements reviews and risk

reduction activities as well as operational test and evaluation of each AWIPS

release are sufficient to ensure that unneeded AWIPS capabilities are

revised or not implemented. In addition, NOAA added its view that

validating capabilities to mission impact is not efficient or cost-effective

and would cause further delay and expense to the program. We disagree.

While we reiterate our commendation for AWIPS requirements reviews and

risk reduction activities to date and support plans to prototype future

system releases, the fact remains that none of these activities were or are

intended to demonstrate the mission impact of AWIPS capabilities.

Page 3 GAO/AIMD-96-74 Weather Forecasting Recommendations
B-271478 

Moreover, waiting until the operational test and evaluation stage of each

AWIPS release is too late to assess mission impact because by this time

considerable sums of money will have already been invested in the

capabilities’ development. Additionally, while validating capabilities to

mission impact would require some resource investment now, it could also

prevent time and money being spent unnecessarily on developing AWIPS

capabilities that promise little in terms of mission improvement. Thus, we

continue to support our recommendation.

NOAA did not comment specifically on our second recommendation to fully

test and correct all material defects in each AWIPS build before software

development associated with a succeeding build begins. However, NOAA

stated that its plans for overlapping AWIPS’ builds does not introduce

unnecessary risks and stated that judicious overlapping is efficient and

appropriate for those builds that do not need to wait for the previous build

to be “frozen.” Further, while acknowledging the potential risks of AWIPS

build overlaps, it noted two factors that mitigate these risks. First, NOAA

stated that there is minimal overlap associated with builds 1 and 2 and that

the two areas in which overlap occurs (data modeling and

communications software) are inherently stable because of early

development and demonstration activities. Second, NOAA stated that

development in the later builds is primarily hydrometeorological

applications and that these applications will not interact directly with each

other. We agree that overlapping builds can be appropriate and efficient,

and that “freezing” one build before moving to the next is not always

efficient. Further, we are not questioning the degree of interaction among

hydrometeorological applications. Our concern is that AWIPS plans call for

extending or adding to baseline applications, such as those that process

radar or satellite products for display on AWIPS monitors, in succeeding

builds before these baselines have even begun formal testing—much less

before the build has been stabilized—by removing all material defects. By

not doing so, NWS runs the risk of introducing defects on top of unresolved

defects. Thus, we continue to support our recommendation that NWS fully

test each AWIPS build and correct all material defects before beginning

software development associated with the next build.

Although NOAA did not directly address our third recommendation to

establish an AWIPS quality assurance program, it agreed that software

quality assurance is important for NWS in-house software development (i.e.,

hydrometeorological applications), and it stated that NWS has hired a

computer specialist for the AWIPS program manager’s staff to oversee the

government’s software development activities. NOAA also stated that NWS is

Page 4 GAO/AIMD-96-74 Weather Forecasting Recommendations
B-271478 

following accepted software development practices and that an

independent review team draft report gives a “generally favorable

impression of the current development methodology.” Despite NWS’ recent

actions, it is important to implement our recommendation for two reasons.

First, hiring a computer specialist falls far short of our recommendation

because such action does not provide either the structure or content of a

disciplined, thorough, or independent quality assurance program. Second,

since NOAA provides no evidence or details of actions taken to follow

“accepted software development practices” and no basis for the “generally

favorable impression” of its development methodology, these points are

unconvincing. Thus, we continue to support our recommendation.

Finally, NOAA disagreed with our fourth recommendation to obtain an

independent assessment of the cost to develop and deploy AWIPS, adding

that it plans to reevaluate its current cost estimate of $525 million in

August 1996 when the deployment phase contract is renegotiated and that

it anticipates no significant increases in the current estimate. NOAA further

stated that developing a new estimate at this time would add little value to

program and budget planning. In defending the validity of its current

estimate of $525 million, NOAA also dismissed each of the omitted cost

items cited in our report for reasons cited below. We disagree that a new

estimate would add little value, particularly in light of expressed interest

by members and staff of the Subcommittee on Energy and Environment,

House Committee on Science, in having an up-to-date, reliable estimate

upon which to base authorization decisions. Also, given NOAA’s dismissal of

all the omitted cost items cited in our report, our recommendation remains

valid. NOAA’s comments on each of the omitted cost items cited in our

report and our response follow.

• NOAA stated that the $9 million increase associated with extending the

deployment phase by 1 year should not be added to the AWIPS project cost

estimate because NWS is committed to completing the deployment phase in

24 months as originally planned. This contradicts NWS documentation

dated February 9, 1996, stating that the AWIPS deployment phase would be

extended 1 year and would result in an additional $9 million, and NOAA has

provided no documentation rescinding this plan or reassessing its cost.

• NOAA repeats its claim that the omission of a $10 million cost increase to

the development phase contract will be offset by a corresponding

decrease in the deployment phase contract. However, as stated in our

report, since the deployment phase contract has yet to be renegotiated,

there is no basis for NOAA’s claim.

Page 5 GAO/AIMD-96-74 Weather Forecasting Recommendations
B-271478 

• NOAA stated that the $2 million omission for recently identified product

improvements should not be included in the AWIPS cost estimate because

this AWIPS enhancement will be developed after the deployment phase is

completed. We disagree. Known system development costs cannot

rationally be excluded from cost estimates. Moreover, the AWIPS project

cost estimate already includes numerous other AWIPS product

improvements and thus excluding the cost of this product improvement

would be inconsistent with NWS’ own practice.

• NOAA acknowledged that the $525 million estimate provides a relatively

small reserve for contract modifications needed to address such things as

code growth. However, NOAA contends that inherent conservatism in the

deployment phase component of the estimate will ultimately produce

more contingency money than is currently apparent in the estimate. Also,

NOAA stated that it does not expect major modifications for the remainder

of the development phase. Despite NOAA’s optimism, it admits that contract

costs (development and deployment) are the greatest variable in the

current cost estimate. Further, prudent software engineering practices

recommend planning for as much as 50 percent growth in software size for

systems at AWIPS’ stage of development. Finally, federal agencies’ past

performance in acquiring major, software intensive systems is routinely

characterized by high cost overruns and significant schedule slippage.

Thus, we find no basis for NOAA to conclude that contingency money will

appear before the deployment phase contract is renegotiated.

The full text of NOAA’s comments is provided in appendix I.

This report contains recommendations to you. The head of a federal

agency is required by 31 U.S.C. 720 to submit a written statement on

actions taken on these recommendations to the Senate Committee on

Governmental Affairs and the House Committee on Government Reform

and Oversight not later than 60 days after the date of this report. A written

statement also must be sent to the House and Senate Committees on

Appropriations with the agency’s first request for appropriations made

more than 60 days after the date of this report.

Page 6 GAO/AIMD-96-74 Weather Forecasting Recommendations
B-271478 

We are sending copies of this letter to each member of the Subcommittee

on Energy and Environment, House Committee on Science, and other

interested parties. We will send copies to others upon request. If you have

questions or wish to discuss the issues in this report further, please

contact me at (202) 512-6240. Major contributors to this report are listed in

appendix II.

Sincerely yours,

Jack L. Brock, Jr.

Director, Information Resources Management/

 Resources, Community, and Economic Development

Page 7 GAO/AIMD-96-74 Weather Forecasting Recommendations
Contents

Letter 1

Appendix I 

Comments From the

National Oceanic and

Atmospheric

Administration

10

Appendix II 

Major Contributors to

This Report

17

Abbreviations

AWIPS Advanced Weather Interactive Processing System

NOAA National Oceanic and Atmospheric Administration

NWS National Weather Service

Page 8 GAO/AIMD-96-74 Weather Forecasting Recommendations
Page 9 GAO/AIMD-96-74 Weather Forecasting Recommendations
Appendix I 

Comments From the National Oceanic and

Atmospheric Administration

Note: GAO comments

supplementing those in the

report text appear at the

end of this appendix.

Page 10 GAO/AIMD-96-74 Weather Forecasting Recommendations
Appendix I 

Comments From the National Oceanic and

Atmospheric Administration

See comment 1.

See comment 1.

Page 11 GAO/AIMD-96-74 Weather Forecasting Recommendations
Appendix I 

Comments From the National Oceanic and

Atmospheric Administration

Page 12 GAO/AIMD-96-74 Weather Forecasting Recommendations
Appendix I 

Comments From the National Oceanic and

Atmospheric Administration

See comment 1.

See comment 1.

Page 13 GAO/AIMD-96-74 Weather Forecasting Recommendations
Appendix I 

Comments From the National Oceanic and

Atmospheric Administration

Page 14 GAO/AIMD-96-74 Weather Forecasting Recommendations
Appendix I 

Comments From the National Oceanic and

Atmospheric Administration

See comment 2.

Page 15 GAO/AIMD-96-74 Weather Forecasting Recommendations
Appendix I 

Comments From the National Oceanic and

Atmospheric Administration

The following are GAO’s comments on the National Oceanic and

Atmospheric Administration’s letter dated April 17, 1996.

GAO Comments 1. Discussed in the “Agency Comments and Our Evaluation” section of the

report.

2. Discussion of this point deleted.

Page 16 GAO/AIMD-96-74 Weather Forecasting Recommendations
Appendix II 

Major Contributors to This Report

Accounting and

Information

Management Division,

Washington, D.C.

Rona B. Stillman, Chief Scientist for Computers and Telecommunications

Randolph C. Hite, Assistant Director

Keith A. Rhodes, Technical Assistant Director

David A. Powner, Evaluator-in-Charge

Robert C. Reining, Information Systems Analyst

Colleen M. Phillips, Information Systems Analyst

Cristina T. Chaplain, Communications Analyst

(511413) Page 17 GAO/AIMD-96-74 Weather Forecasting Recommendations
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100