Department of Energy: Poor Management of Nuclear Materials Tracking System Makes Success Unlikely
Released: 1995-09-05T00:00:00

Letter

Background

NMMSS is the United States’ official nuclear materials tracking and accounting system. NMMSS provides information on nuclear materials to support both domestic programs and international nuclear policies. Keeping track of the growing amount of nuclear materials is especially important as a result of the breakdown of the Soviet Union and increases in both domestic and international terrorism. Tracking and accounting for the hundreds of tons of plutonium, highly enriched uranium, and other nuclear materials that have accumulated are important to help (1) ensure that nuclear materials are used only for peaceful purposes, (2) protect nuclear materials from loss, theft, or other diversion, (3) comply with international treaty obligations, and (4) provide data to policymakers and other government officials on the amount and location of nuclear materials.

The NMMSS database contains data on nuclear materials supplied and controlled under international agreements, including U.S.-supplied international nuclear materials transactions, foreign contracts, import/export licenses, government-to-government approvals, and other DOE authorizations, such as authorizations to retransfer U.S.-supplied materials between foreign countries. NMMSS also maintains and provides DOE with information on domestic production and materials management, safeguards, physical accountability, financial and cost accounting, and other data related to nuclear materials accountability and safeguards for Nuclear Regulatory Commission licensees.

DOE and the Nuclear Regulatory Commission cosponsor NMMSS, and it is managed and operated by a DOE contractor—Martin Marietta Energy Systems, Incorporated. NMMSS has been used to account for U.S. imports and exports of nuclear materials since 1977.

Because the existing NMMSS is an older system, DOE decided to replace and modernize it. The existing NMMSS is housed on a mainframe using unstructured COBOL code. Performing modifications on the existing NMMSS and designing custom reports are difficult because of the volume and complexity of the code. Accordingly, DOE’s Office of Nonproliferation and National Security, which is responsible for operating NMMSS, tasked the Lawrence Livermore National Laboratory with developing a new NMMSS. Livermore hired a subcontractor to perform this task in February 1994 and assigned a program manager to oversee the effort. In April 1994, Livermore formed a technical advisory committee, composed of senior computer scientists and material control and accountability specialists, to assist the program manager in overseeing the system development.

The replacement NMMSS is scheduled to become operational on September 1, 1995. Martin Marietta is scheduled to discontinue operation of the existing NMMSS during September 1995.

Scope and Methodology

To address our objectives, we reviewed the replacement NMMSS contract, transition plan, test plan, and various other draft system documents. We requested documentation on the status of the system coding and testing; however, none was available for our review. We also analyzed documentation provided to us by Lawrence Livermore’s technical advisory committee on the subcontractor’s development efforts. In addition, we analyzed documentation from various user groups on their concerns with the NMMSS development. We analyzed DOE Order 1330.1D, Computer Software Management, to determine its applicability to this project and whether or not it was being followed.

We interviewed DOE officials in the Office of Nonproliferation and National Security concerning actions taken to implement the recommendations in our previous report and the status of the NMMSS development. We also interviewed the NMMSS program manager, members of Lawrence Livermore’s technical advisory committee, contract officials at DOE and Lawrence Livermore, and the NMMSS subcontractor’s lead programmers, system engineer, and project managers to determine the status of the system development. In addition, we interviewed officials in DOE’s Defense Programs Office—the biggest user of NMMSS information—on their concerns about the replacement NMMSS development.

We performed our work between February 1995 and May 1995, in accordance with generally accepted government auditing standards. Our work was primarily done at DOE’s headquarters in Washington, D.C., and its offices in Germantown, Maryland; at Lawrence Livermore National Laboratory in Livermore, California; and at the subcontractor’s facility in Norcross, Georgia. The Department of Energy provided written comments on a draft of this report. These comments are presented and evaluated in the report, and are reprinted in appendix I.

DOE Is Pursuing the Replacement NMMSS Without Addressing Critical Planning Issues

In December 1994, we reported that DOE did not adequately plan the development effort for the replacement NMMSS. For example, DOE did not follow sound system development practices such as identifying and defining users’ needs and adequately exploring design alternatives that would best satisfy these needs in the most economic fashion. Accordingly, we recommended that DOE determine users’ requirements, investigate alternatives, conduct cost-benefit analyses, and develop a plan to meet identified needs before investing further resources in the replacement NMMSS.

In its official response to the recommendations in our prior report, DOE stated that it did not concur with our recommendations and that it did not believe it would be cost-effective to delay its effort to transition from the existing system to the new system. Further, in commenting on a draft of this report, the Acting Director of the Office of Nonproliferation and National Security stated that DOE’s planning was sufficient. However, because of DOE’s lack of basic planning, it does not know if the system will fulfill the needs of its major users or be cost-effective.

System Development Risks Are High

Adhering to generally accepted system development practices helps to ensure that information systems perform as desired. These practices include (1) generating clear, complete, and accurate documentation throughout the system development process, (2) placing the software development under configuration management, and (3) ensuring that the system successfully completes acceptance testing prior to becoming operational. However, because DOE has not required the subcontractor to follow any of these practices for the replacement NMMSS, the Department does not know how much of the system development is completed and whether the part that is completed performs as required. As a result, the risk of system failure is inordinately high.

Little System Development Documentation Exists

DOE Order 1330.1D, Computer Software Management, requires that the development of a system be documented so that, among other things, the status of the system is known at all times. Documentation, such as the results of system testing and the tracking of source code as it changes, allows program managers to review the development’s progress and determine if requirements are being met.

The subcontractor developing the replacement NMMSS could not provide any system documentation—software specifications, system requirements, results of formal reviews (e.g., system/preliminary/critical design) or informal system testing reports, operational procedures, quality assurance checklists, or project tracking reports. Because little system documentation exists, and the contract does not require any interim deliverables that measure system performance, DOE does not know the status of the system development. In addition, members of Livermore’s technical advisory committee told us they have been unable to obtain the documentation they needed to determine the status of the development effort. As a result, the committee said it could not accurately determine such factors as the number of lines of code in the system. In fact, the advisory committee could only estimate system size in very gross terms—between 10,000 and 100,000 lines of code.

DOE officials agreed that the development effort is largely undocumented and stated that DOE historically has not enforced its own regulations requiring system documentation. At the conclusion of our review and in commenting on a draft of this report, DOE officials told us that they will begin to require such documentation for the replacement NMMSS.

Configuration Management Was Not Used

A successful system development project should include a software configuration management plan that clearly defines the procedures for identifying, accounting for, and reporting on changes to software items that are under configuration control. Configuration management is necessary throughout the life cycle of a software project because it provides (1) a control mechanism to ensure that the software status is accurately known at all times and (2) a baseline for system developers and testers.

Although the subcontractor developed a software configuration management plan for the replacement NMMSS, no software had been placed under configuration control. As a result, DOE does not know what version of the software is current, which versions of the software have been tested, what problems were identified during testing, and what corrections are being made. Developing software without configuration management frequently results in projects that are delivered late, exceed budget, and perform poorly.

Officials in DOE’s Office of Nonproliferation and National Security agreed that the replacement NMMSS software had not been placed under configuration management at the time of our exit conference. The officials stated that, until recently, they had not required the use of configuration management on software development projects. In its written comments on a draft of this report, DOE stated that the replacement NMMSS is now being placed under configuration control.

DOE Does Not Plan to Adequately Test NMMSS

During acceptance testing, tests are performed to determine if a system will meet its hardware, software, performance, and user operational requirements. Acceptance testing is usually performed by the system developer and witnessed by an independent verification and validation group, which includes system users. Such testing is important to determine if the new system performs as required.

The previous implementation schedule for the replacement NMMSS called for acceptance testing and 2 months of parallel operation with the existing NMMSS. In addition, in a January 1994 memorandum, an official from DOE’s Office of Nonproliferation and National Security stated that the replacement NMMSS would not be made operational until “it has been demonstrated that the new system is capable of meeting present and new customer needs and requirements.” Adhering to this position on testing the replacement NMMSS would have greatly reduced system risks.

In January 1995, DOE changed its position and decided to make the replacement NMMSS operational without performing acceptance testing. DOE officials stated that this decision was made to avoid the cost of simultaneously funding both the existing and replacement systems. Instead, DOE plans to perform what it is calling “system testing” on a subset of system reports—87 of approximately 500 reports. While DOE stated that these 87 reports were selected based on users’ needs, it could not produce documentation to validate this statement.

The only system testing at the time of our review was the informal testing that the subcontractor stated it had performed. However, the subcontractor could not provide documentation on either its test plans or the test results. In its written comments on a draft of this report, DOE officials stated that system test procedures have now been written and approved.

In addition, parallel operations with the existing NMMSS are not scheduled. During parallel operations, both systems would perform all required functions, and then outputs would be compared to ensure that the replacement system is producing accurate reports. Because the replacement NMMSS will replicate the functions of the existing NMMSS, a period of parallel operations is especially important.

Without parallel processing, DOE is introducing additional risk that the replacement system will not perform all functions of the existing system or, more importantly, that the information produced by the replacement system will not be accurate. As a result, DOE cannot guarantee its users that the information they need from NMMSS to do their jobs will continue to be available. NMMSS users told us that information they get from the existing NMMSS within hours could take weeks or months to gather if they cannot obtain it from the new NMMSS or if they cannot be sure that the information in the new NMMSS is accurate.

Conclusions

DOE has stated that it will discontinue the existing system on September 1, 1995, and begin operation of the replacement NMMSS without acceptance testing. However, DOE’s replacement NMMSS is being developed in an undisciplined, poorly controlled manner that makes success unlikely. Planning was inadequate and basic system development practices are not being followed. As a result, DOE will not know if the replacement NMMSS will produce the accurate and timely reports needed to meet users’ needs before it accepts the system and pays the subcontractor.

DOE’s disregard for basic system development practices necessary to ensure the accuracy and dependability of its nuclear tracking system is inconsistent with the importance of NMMSS, which provides the United States’ official record for tracking nuclear materials. It is not in DOE’s best interests, therefore, to disconnect the existing NMMSS and replace it with an untested, undocumented new system. The history of software development is littered with systems that failed under similar circumstances.

Recommendations

We recommend that the Secretary of Energy immediately terminate any further development of the replacement NMMSS. Further, as we recommended in our December 1994 report, the Secretary should direct the Office of Nonproliferation and National Security to determine users’ requirements, investigate alternatives, and conduct cost-benefit analyses before proceeding with any plan to develop a replacement NMMSS.

If, after thorough planning, the Office proceeds with plans to develop a new NMMSS, we recommend that it follow generally accepted system development practices. In the interim, we recommend that DOE continue using the existing NMMSS system until the above recommendations are addressed.

Agency Comments and Our Evaluation

The Department of Energy provided written comments on a draft of this report. Their comments are summarized below and reproduced in appendix I.

The Department of Energy agreed with the need for systems development documentation, configuration management, and adequate testing. However, the Department did not concur with our assessment of its analyses and planning for the system development effort, or with our recommendation that it terminate the system development until users’ requirements, alternatives, and cost-benefit analyses have been performed. DOE stated that its planning was adequate because it is converting an existing system from an unstructured language to a structured, fourth generation language, rather than developing a new system.

We disagree. Without sound analyses or planning, DOE does not know that “converting an existing system” is a cost-effective way to meet its needs. Furthermore, as our report discusses, DOE is implementing this unsupported approach in an unsatisfactory manner. Therefore, DOE should discontinue its current effort and perform users’ requirements, alternatives, and cost-benefit analyses before proceeding.

As arranged with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the date of this letter. At that time, we will provide copies of this report to the Secretary of Energy; the Director, Office of Management and Budget; appropriate congressional committees; and other interested parties. Copies will also be made available to others upon request.

Please call me at (202) 512-6253 if you or your staff have any questions. Major contributors to this report are listed in appendix II.

Comments From the Department of Energy

Major Contributors to This Report

Accounting and Information Management Division, Washington, D.C.

Valerie C. Melvin, Assistant Director Keith A. Rhodes, Technical Assistant Director Suzanne M. Burns, Evaluator-in-Charge Linda J. Lambert, Senior Auditor The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (301) 258-4097 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

Address Correction Requested