Information Security: National Nuclear Security Administration Needs to Improve Contingency Planning for Its Classified Supercomputing Operations
Released: 2010-12-09T00:00:00

Letter

Background

NNSA relies on its Stockpile Stewardship Program to ensure the safety, security, and effectiveness of the nuclear weapons stockpile. The Stockpile Stewardship Program is comprised of various elements, including, but not limited to: (1) the Advanced Simulation and Computing (ASC) Campaign, which provides the computational science and simulation tools to understand the behaviors and effects of nuclear weapons; (2) Directed Stockpile Work, which provides evidence of the health of the nuclear weapons stockpile and involves day-to-day maintenance of these weapons, including life extension efforts; (3) the Science Campaign, which provides tools and capabilities geared toward advancing the general understanding of all nuclear weapons systems; and (4) the Engineering Campaign, which provides a sustained basis for stockpile certification and assessments throughout the life cycle of each weapon. The coordination among the Stockpile Stewardship elements is instrumental to increasing NNSA’s confidence in the performance of nuclear weapons.

To help accomplish its Stockpile Stewardship mission, NNSA relies on the three weapons laboratories—Los Alamos, Livermore, and Sandia. Los Alamos and Livermore are the two design laboratories that are responsible for designing the nuclear weapons’ explosive package and conducting research to better understand nuclear weapons phenomena. Sandia is an engineering laboratory and has principal responsibility for the research, design, and development of nonnuclear warhead components; integration of these components with Los Alamos and Livermore; and overall warhead systems integration with DOD. In accordance with NNSA, management and operations contractors, who are responsible for day-to-day operations of the laboratories, are required to adhere to agency policies.

At the time of our review, NNSA’s classified supercomputing resources consisted of 12 classified supercomputing systems. Figure 1 shows the hows the hardware configuration of a supercomputing system. hardware configuration of a supercomputing system.

NNSA classified supercomputing systems employ a large number of interdependent processors, which are the core unit of a computer that gathers instructions and data. These processors are mounted onto a compute chip, which is the portion of the system that carries out the instructions of a computer program. These compute chips are inserted onto a compute card, which also holds memory for the compute chips to use. A number of compute cards are attached to a node card, which have one or more processors with a common memory and are connected by high-speed interconnection networks. Each node card is inserted into a single cabinet, and that configuration is repeated many times to build a single supercomputing system. Each supercomputing system has a peak performance, which is the maximum rate of floating-point operations per second (FLOPS) that the system can sustain. Currently, almost all NNSA classified supercomputer systems operate at the teraFLOP level, which represents a trillion FLOPS.

According to NNSA, the laboratories have three types of classified supercomputing systems: Capacity: Small systems that execute parallel problems with more modest computational requirements. These systems serve as the workhorse for the ASC program and are responsible for processing the day-to-day supercomputing workload.

Capability: This type of supercomputer is used to solve the largest and most demanding problems that other computing systems cannot manage.

Advanced architecture: Research and development systems that assist the ASC program in preparing to rapidly deploy and exploit the next generation of supercomputing technology. These systems have a targeted workload and serve as the foundation for the next generation of NNSA supercomputers.

Table 1 shows the classified supercomputing systems currently in use at the three weapons laboratories.

NNSA’s classified supercomputing capabilities consist of supporting resources, including (1) parallel files systems, which store transitory data; (2) network file systems, which store user and project data for a calculation; (3) archival storage systems, which serve as storage for data; and (4) visualization systems, which enable users to better comprehend the results of their computations.

NNSA’s classified supercomputing systems are connected via its Enterprise Secure Network and the Distance Computing (DISCOM) network, which function as supporting resources for the classified supercomputing environment. The Enterprise Secure Network provides classified communications across the nuclear weapons complex, including security services and other activities that ensure the flow of NNSA’s data sharing and business missions. DISCOM provides secure, high-speed remote access for intra- and inter-site file transfers and enables users, across the three weapons laboratories, to operate on remote computing resources as if they were local. DISCOM and the Enterprise Secure Network serve as the backup networks to each other. Figure 2 shows the composition of NNSA’s classified supercomputing network infrastructure.

NNSA reported obligating approximately $1.7 billion from fiscal years 2007 through 2009 to support ASC program activities at the three weapons laboratories. The $1.7 billion was predominantly associated with three efforts: Weapons codes and models. This effort is intended to develop and improve weapons simulation models and codes for predicting the behavior of weapons systems and devices in the nuclear stockpile.

Computational systems and software environment. This effort is intended to provide ASC users a stable, seamless computing environment for ASC-deployed platforms. It is responsible for procuring, delivering, and deploying ASC computational systems and user environments via technology development and integration across the three weapons laboratories.

Facility operations and user support. This effort is intended to provide both the necessary physical facility and operational support for reliable supercomputing and storage environments, as well as a suite of user services for effective use of the three weapons laboratories’ computing resources. Facility operations cover physical space, power and other utility infrastructure, and local- and wide-area networking, as well as system administration, cyber security, and operations services for ongoing support. The user support function includes planning, development, integration and deployment, continuing product support, and quality and reliability activity collaborations.

To strengthen the security of information and information systems across the federal government, including those at NNSA’s weapons laboratories, the Federal Information Security Management Act of 2002 (FISMA) requires each agency to develop, document, and implement an agencywide information security program that supports the operations and assets of the agency, including those provided or managed by another agency or contractor on its behalf. This security program is to include plans and procedures to ensure the continuity of operations for information systems that support the agency’s operations. Pursuant to its FISMA responsibilities, the National Institute of Standards and Technology (NIST) has issued federal standards and guidelines on information security, such as a contingency planning guide for federal information systems, and recommended security controls, which address contingency and disaster recovery planning and testing. To further ensure the security of national security systems, the Committee on National Security Systems (CNSS) requires federal agencies with national security systems to implement a comprehensive set of security controls and enhancements for these systems. CNSS requires that each agency implement a contingency and disaster recovery planning capability that ensures the integrity and availability of its national security information and information systems.

FISMA, NIST guidelines, and CNSS policies all call for contingency and disaster recovery planning—also referred to as continuity of operations for information systems—for critical components of information protection. DOE and NNSA policies also regard contingency and disaster recovery plans as being necessary for information protection. If normal operations are interrupted, contingency and disaster recovery plans allow senior agency officials to detect, mitigate, and recover operations. Examples of the key components that make up contingency and disaster recovery planning programs include (1) assessing the criticality and sensitivity of computerized operations and identification of supporting resources such as developing business impact analyses (BIA), (2) taking steps to prevent and minimize potential damage and interruption such as establishing data backup processes, (3) developing comprehensive contingency and disaster recovery plans, and (4) conducting periodic testing of contingency and disaster plans.

The extent to which controls—such as contingency and disaster recovery planning—are implemented depends on a level of risk assigned to the system or information maintained on the system. NIST standards and guidelines, CNSS instructions, and NNSA policy allow consideration of risk in determining the level of protection of systems and data. These standards and policies require that organizations consider the impact or consequences of loss as it relates to the confidentiality, integrity, and availability of the information, and assign a value of low, moderate, or high impact levels. For contingency and disaster recovery planning, consideration of “availability” is the key element. NNSA policy defines the values for the consequences of loss associated with availability as follows: High—Loss of life might result from loss of availability; information must always be available on request, with no tolerance for delay; loss of availability will have an adverse effect on national-level interests; federal requirement (i.e., requirement for material control and accountability inventory); or loss of availability will have an adverse effect on confidentiality.

Moderate—Information must be readily available with minimum tolerance for delay; bodily injury might result from loss of availability; or loss of availability will have an adverse effect on organizational-level interests.

Low—Information must be available with flexible tolerance for delay.

NNSA Has Not Fully Implemented Contingency and Disaster Recovery Planning and Testing for Its Classified Supercomputing Assets

Contingency and disaster recovery planning and testing for NNSA’s classified supercomputing systems have not been fully implemented at each of the three weapons laboratories—Los Alamos, Sandia, and Livermore. Specifically, NNSA did not ensure that the laboratories (1) developed BIAs to determine the impact of potential service disruptions, (2) fully tested data backup processes, and (3) developed and tested contingency and disaster recovery plans. These shortcomings existed, at least in part, because NNSA’s component organizations were unclear of their roles and responsibilities for providing oversight in the laboratories’ implementation of contingency and disaster recovery planning. Until the agency fully implements a contingency and disaster recovery planning program for its classified supercomputing assets at the weapons laboratories, it has limited assurance that vital information can be recovered and made available to meet national security priorities and requirements.

Not All of the Laboratories Assessed the Criticality and Sensitivity of Supercomputer Operations and Resources, or Potential Outage Impact

To assess the criticality and sensitivity of computerized operations and identification of supporting resources, NIST guidelines state that agencies should determine their recovery strategies by performing business impact analyses of their systems. A BIA is an analysis of information technology system requirements, processes, and interdependencies used to characterize system contingency requirements and priorities in the event of a significant disruption. NIST guidelines state that agencies conduct a BIA to identify critical information systems to fully characterize the system’s requirements, processes, and interdependencies to determine contingency requirements and priorities. In addition, according to NIST guidelines, the BIA process should follow three main steps: (1) identify critical data and information technology resources, (2) identify outage impacts and allowable outage times, and (3) develop recovery priorities and strategies. NNSA policy also requires a BIA to identify systems that provide critical services to site operations and prioritize these systems and their components.

One of the laboratories—Los Alamos—had conducted a BIA that addressed its classified supercomputing systems, generally following the three steps of a BIA. However, the BIA was not always specific. For example, the laboratory identified critical information technology resources for each of its classified supercomputing systems, but did not specifically identify the critical data. Instead, Los Alamos noted that the systems are not considered mission critical nor mission essential to the business needs of the laboratory, and that the consequence of loss for system availability is low. Additionally, it defined a specific number of days for the allowable time frames for fully and partially disabled systems, but did not provide specifics on allowable outage impacts. Further, the analyses indicated high-level recovery priorities, but did not provide specifics regarding the recovery process or strategies that would be used for recovery efforts.

The other two laboratories did not conduct BIAs specifically for classified supercomputing systems, but plan to do so. Livermore has a BIA in place for its logical assets—the applications and services that provide basic operational support to the Livermore computing environment, but the BIA did not address any of the classified supercomputing systems. However, at the time of our site visit, Livermore officials stated they were beginning the process of developing a BIA that would address their information technology needs for their classified supercomputing systems, but the process was still in the planning stage. Similarly, according to Sandia officials, the laboratory has BIAs that address its unclassified information technology systems, but does not currently have one specifically for its classified supercomputing systems. However, Sandia officials indicated that they plan to conduct a BIA for classified supercomputing systems in 2011.

Although the two laboratories have not conducted any BIAs—in line with the BIA conducted by Los Alamos—they have considered the risk of consequence of loss from availability as low impact. NNSA also considers the consequence of loss as low impact. In addition, NNSA and the three laboratories do not consider the classified supercomputers to be “mission critical.” One laboratory categorized the systems as “mission essential,” while another referred to them as “mission support elements, not mission essential elements.” However, NNSA’s mission includes maintaining the safety, security, and effectiveness of the nuclear deterrent without nuclear testing. The supercomputers provide a necessary means to determine the effects of changes to current weapons systems and to determine a level of confidence in the performance of future untested systems. The classified supercomputing capabilities serve as the computational surrogate to nuclear weapons testing and are central to national security.

Regarding recovery priorities and strategies, each of the laboratories indicated that it would likely rely on a process that is currently being used for the capability system shared among the laboratories. The laboratories generally rely on the Capability Computing Campaign to prioritize the workload and develop priorities for jobs that need to be run on the capability system. In the event of a service disruption or emergency, laboratory officials told us that they would likely rely on the same process for all of their systems. However, this process has not been documented as a means for establishing overall recovery priorities across the laboratories.

Until all of the laboratories have a BIA in place for their classified supercomputing systems that (1) identifies and categorizes critical data, (2) identifies acceptable allowable outage impacts and time frames, and (3) establishes emergency processing priorities and strategies, the potential impact of a system outage will remain unclear.

The Laboratories Have Backup Processes in Place, but One Storage Site May Be Susceptible to Damage

Data backup processes offer a means of taking steps to prevent and minimize potential damage and interruption to computerized services. NIST guidelines, as well as CNSS instructions and NNSA policies, call for agencies to conduct backups of user-level information, system-level information, and information system documentation. In addition, NIST, CNSS, and NNSA all provide that agencies establish an alternate storage site that is separated from the primary storage site so that both are not susceptible to the same hazards. To ensure the availability of data stored in the alternate storage site, NIST and CNSS require that agencies test the backup information to verify the integrity of the data.

All of the laboratories had backup processes in place. Each of the laboratories follows similar data backup processing—both manual and automated procedures—to back up user-level information, system-level information, and information system documentation. For example, this information can include global directories, user home directories, project directories, desktop systems, and critical systems documentation. Backups occur in increments: daily incremental backups to disk, weekly full backups to tape, and monthly full-system backups to tape (with a 6- month on-site storage retention policy). The laboratories also have vendor- provided software that takes periodic snapshots of user directories for storage retention purposes. The snapshot process can be performed manually or can be set up for automatic processing. Users are encouraged to maintain their data in a shared environment on the network and are allowed to make their own determinations regarding what data should be backed up from the classified supercomputing systems.

Not all of the laboratories have an alternate storage site sufficiently separated from the primary site to not be susceptible to the same hazards. Two of the three laboratories have alternate storage sites a considerable distance from their primary storage site. Livermore sends its system backups electronically to Los Alamos every 6 months. Sandia sends its backup data to its alternate site locations (e.g, the California site sends its data to the New Mexico site and the New Mexico site sends its data to the California site). However, Los Alamos maintains its alternate storage facility on-site in a building located less than 1 mile away from the primary local backup storage facility. Consequently, both sites could be susceptible to the same hazards, such as a wildfire.

The laboratories had processes in place to verify the integrity of the backed up data. However, tests of their backup procedures rely predominantly on ad hoc recovery, rather than periodically planned tests. Los Alamos officials indicated that thousands of file recoveries have been performed over the years by end users as part of their testing. Livermore officials stated that the laboratory tests its local backup procedures through actual system usage on almost a daily basis, and tests their remote backup procedures at least once annually. Further, Sandia officials told us they had successfully tested a sample of data at their offsite facility.

Not All Laboratories Had Developed and Tested Contingency and Disaster Recovery Plans

NIST guidelines and CNSS policies call for the development and testing of contingency plans and the development of disaster recovery plans for each information system to ensure that, in the event of a service disruption, the work and supporting functions of the agency can continue to be performed. According to NIST guidelines, at a minimum, the contingency plan should address the identification and notification of key personnel, plan activation, system recovery, and system reconstitution to meet the needs of the agency’s critical supporting operations. The guidelines also state that the plan should be tested periodically; CNSS specifies that the frequency of testing should be annually. NIST also notes that the disaster recovery plan should be designed to restore operability of the targeted system, application, or computer facility at an alternate site after a major service disruption. DOE and NNSA policies also require the development of contingency and disaster recovery plans and the testing of these plans in line with NIST and CNSS.

Each of the laboratories had developed contingency plans for their classified supercomputing systems; however, the plans were not always comprehensive, and at the time of our site visits, these plans had not been tested. The laboratories addressed disaster recovery planning to a limited extent; none specifically addressed the supercomputing environment. For example, Two laboratories—Los Alamos and Sandia—had contingency plans that addressed the classified supercomputing systems. Although Livermore had an information technology contingency plan and a master security plan, neither specifically addressed the supercomputers. In addition, the plans for both Los Alamos and Sandia included key components such as the identification and notification of key personnel, plan activation, system recovery, and system reconstitution procedures; however, the sufficiency of the level of detail varied. For instance, one plan provided specific details regarding system recovery processes and the notification and identification of key personnel, but provided limited details regarding plan activation and system reconstitution procedures.

At the time of our site visits, none of the laboratories had tested their contingency plans, which were less than a year old. One of the laboratories—Los Alamos—had created testing guides but had not yet conducted formal testing. Subsequent to our site visit, Los Alamos officials indicated that the first test of their plan took place in September 2010 and noted that the results would be finalized in December. Additionally, although Sandia had a contingency plan in place, the plan states that testing is not required because, in the event of a service disruption, the laboratory would either wait until the equipment was fully operational or simply acquire new equipment. This is contrary to NIST guidelines, CNSS instructions, and DOE and NNSA policies.

Each of the laboratories had addressed disaster recovery planning to a limited extent. For example, Los Alamos included disaster recovery planning as a section within their classified supercomputing system’s contingency plans. Although it provided high-level instructions such as directing individuals to call 911 for all emergencies, it did not include information regarding the specifics for restoring operability of the classified supercomputing system at an alternate site after a major service disruption.

None of the plans submitted by Livermore specifically addressed the supercomputing environment, although in the disaster recovery section of the master security plan, the laboratory noted that it had no mission-essential systems in the computing environment, and systems may be offline for an extended period for system upgrades.

Sandia also had disaster recovery plans in place, designed for emergency preparedness and disease response planning needs for the laboratory. These plans focused on emergencies involving the facilities, operations, and activities for the laboratory, and provided individuals with emergency information should pandemics plague the laboratory. However, these plans did not include any information regarding the classified supercomputing systems.

Unless each of the laboratories develops and sufficiently tests comprehensive contingency and disaster recovery plans in accordance with applicable policies and guidance for their classified supercomputing systems, they face a risk of not being able to successfully recover their supercomputing assets and operations after a service disruption.

NNSA Component Organizations Were Unclear of Their Roles and Responsibilities for Providing Oversight

The aforementioned shortcomings existed, at least in part, because NNSA’s component organizations were unclear of their roles and responsibilities for providing oversight in the laboratories’ implementation of contingency and disaster recovery planning. FISMA requires that the chief information officer, in coordination with other senior agency officials, manage the development and implementation of an agencywide information security program that includes plans and procedures to ensure continuity of operations for information systems that support the operations and assets of the agency. NIST guidelines and DOE policies call for individuals with information system or security management and oversight responsibilities to take responsibility for the development, implementation, assessment, monitoring, reviewing, and updating of security planning policies and procedures, which includes contingency and disaster recovery plans. Further, the NNSA Safety Management Function and Responsibilities and Authorities Manual states that the chief information officer is responsible for information technology programs and initiatives and for ensuring the security of the agency’s information and systems.

Although roles and responsibilities are defined at a high level in FISMA, NIST guidelines, as well as DOE and NNSA policies, NNSA component organizations were confused about their roles in providing oversight of the laboratories’ implementation of contingency and disaster recovery planning for the supercomputing systems. For example, at the beginning of our review, ASC officials told us that, although they were responsible for administering and managing the program that uses the classified supercomputing systems, they were not responsible for contingency and disaster recovery planning. Instead, they directed us to the Office of the Chief Information Officer (OCIO), where officials told us that they were not responsible for contingency and disaster recovery planning for these systems, and noted that they would only provide guidance if requested by ASC. Further, OCIO officials told us that ASC has not requested any assistance. ASC officials subsequently acknowledged that they had responsibility for contingency and disaster recovery planning; however, this organizational responsibility is contrary to NIST guidelines and DOE policies, as well as NNSA’s own manual, which gives this responsibility to the OCIO. In the absence of effective oversight, the laboratories did not consistently comply with, or fully implement, federal requirements and guidance related to contingency planning and disaster recovery. Until NNSA clearly establishes and carries out defined roles and responsibilities for OCIO and ASC pertaining to contingency and disaster recovery planning for the classified supercomputing environment, it will not be able to effectively manage and oversee the recovery of its supercomputing operations should service disruptions occur.

The Laboratories Have the Ability to Share Supercomputing Capacity, but Barriers Exist

Technologically, the weapons laboratories have demonstrated the ability to share classified supercomputing capacity using their capacity and capability systems under normal operating conditions. Although these supercomputers process unique workloads and operate independently, they are designed with a similar operating system, resource manager, and job scheduler, which is built on a LINUX foundation. These supercomputers also include application codes that are portable across supercomputing systems and a data network, which allows authorized users local and remote access to the systems.

Although the weapons laboratories have the ability to share supercomputing capacity, barriers exist. One barrier to sharing supercomputing capacity is that the weapons laboratories do not know the minimum supercomputing capacity needed to achieve processing priorities in the event of a service disruption. NIST guidelines recommend, and NNSA policy requires, that capacity planning be conducted so that there is adequate capacity for information processing and supporting resources during contingency operations. Although the weapons laboratories have identified the supercomputing processing needed for normal business operations, they have not identified the minimum supercomputing capacity needed to achieve processing priorities in the event of a service disruption.

Another barrier to sharing supercomputing processing is the disparity in usable supercomputing processing across the laboratories. Figure 3 depicts this disparity by identifying the amount of total usable supercomputing capacity, in teraFLOPS, for each of the three weapons laboratories for 2010 and 2011.

For example, in 2010, total usable capacity at Livermore has been 311 teraFLOPS, whereas Los Alamos and Sandia have had 127 teraFLOPS and 76 teraFLOPS, respectively. Should Livermore experience service disruptions for a sustained amount of time, neither Los Alamos nor Sandia possesses the necessary usable supercomputing capacity to accommodate the additional workload and NNSA will have to reprioritize the computational workloads across the other two laboratories. As previously noted, officials at the laboratories told us that, should disruptions occur, they would use the Capability Computing Campaign model for re-prioritizing the workload. However, this process has not been documented for recovery activities.

Further limiting the ability of the weapons laboratories to recover from a service disruption, in 2011, there will be a significant disparity in projected usable supercomputing capacity. For example, in 2011, Los Alamos’ usable capacity is projected to be 1,427 teraFLOPS, whereas usable capacity at Livermore and Sandia is to be 218 teraFLOPS and 76 teraFLOPS, respectively. Should Los Alamos’ supercomputing systems become unavailable for an extended period of time, neither Livermore nor S possesses sufficient usable supercomputing capacity to achieve its workload and accommodate the additional potential computation workload from Los Alamos. According to laboratory officials, an additional supercomputer will be deployed at Los Alamos in 2011. This supercomputer is a replacement for the single capability supercomputer currently at Livermore that was retired in 2010. Therefore, a significan amount of usable supercomputing capacity will be centralized at Los Alamos. Because the weapons laboratories have not determined the minimum supercomputing capacity requirements for their emerge processing priorities, they may not be able to meet the minimum computation al workload required to meet Stockpile Stewardship milestones.

Another barrier to sharing supercomputing capacity across the weap laboratories is that the capability to share usable capacity on an “on- demand” basis has not been fully tested in a recovery scenario. According to officials at the laboratories, during normal operating conditions, simulation programs have run on other supercomputing systems. However, consideration has not been given to include and test these abilities in a disaster recovery scenario should a service disru occur. As a result, NNSA has limited assurance that its disaster recov approach would work effectively should a service disruption occur.

NNSA Does Not Track the Costs for Ensu ring Contingency and Disaster Recove Planning for Its Superco Assets

Although NNSA reported obligating approximately $1.7 billion from fisc 2007 through 2009 to implement its ASC program activities at the three weapons laboratories, the costs for ensuring the recovery of its classifie supercomputing operations are unknown. Under GAO’s Standards for Internal Control in the Federal Government, financial information sh be recorded and communicated to program managers who need this information to make operational decisions and to effectively allocate resources for program activities.

NNSA officials reported obligating approximately $390 million for facility operations and user support activities, which include the funds associated with contingency and disaster recovery planning activities, but they w unable to provide detailed financial information for contingency and disaster recovery planning activities. According to NNSA officials, costs for contingency and disaster recovery planning for classified supercomputing systems are unknown because ASC program expenditures are part of the NNSA ASC operational budget, whose costs are tracked at an aggregate level. As a result, neither NNSA nor the three weapons laboratories can track what has been spent since fiscal year 20 for ensuring the recovery of classified supercomputing operations and, consequently, they do not know whether funding levels for these activit have been adequate. Although certain components of contingency and disaster recovery planning are in place at the three weapons laborator NNSA is uncertain as to what funds were spent on these information protection activities.

Furthermore, NNSA has not developed contingency planning and disaster recovery cost estimates for its classified supercomputing assets. For fiscal years 2011 through 2014, NNSA projects that it needs about $2.2 billion to implement its ASC activities, which support its Stockpile Stewardship program—$984 million for weapons codes and models, $604 million for computational systems and software environment, and $588 million for facility operations and user support. Although NNSA has developed its out-year funding needs over the next 4 years, it has not developed estimates regarding the future costs for ensuring the recovery of its classified supercomputing assets in the event of service disruptions. Until NNSA develops a means for tracking current contingency and disaster recovery costs and for developing estimates of future costs, the agency will not have the information needed to determine whether it is meeting its goals for effective stewardship of public resources.

Conclusions

All three NNSA weapons laboratories have implemented some components of a contingency planning and disaster recovery program. NNSA, however, has not provided effective oversight to ensure that the laboratories have comprehensive and effective contingency and disaster recovery planning and testing. For example, B IAs that identify critical resources and outage impacts have not been developed for all classified supercomputing systems and existing contingency plans at the laboratories have not been thoroughly tested. Although one laboratory’s analysis is not comprehensive and the other two laboratories have not completed a BIA, NNSA and the laboratories consider the consequence of loss of availability of the classified supercomputers as a low-risk impact, and do not consider them to be mission critical. However, it is unclear how NNSA made this determination given that (1) the analyses have n been completed; (2) NNSA’s mission includes maintaining the safety security, and effectiveness of the nuclear deterrent without nucleartesting; (3) the classified supercomputing capabilities serve as the computational surrogate to underground nuclear weapons testing and are central to our national security; and (4) NNSA has obligated about $1.7 billion over 3 fiscal years to support the Advanced Simulation and Computing program, which includes classified supercomputing activities.

Beyond the activities undertaken by the laboratories, NNSA has not developed a means for identifying, tracking, or re-prioritizing the classified supercomputing workload across the operating environment. In addit the laboratories have not tested offsite recovery capabilities and the agency has not tested the laboratories’ ability to share “on-demand” capacity if needed or determined the minimum capacity needed to meet Stockpile Stewardship Program requirements, particularly in the event that it may need to establish emergency processing priorities. Furt her, although over a billion dollars have been obligated to support the classified supercomputing capabilities within the last 3 years, NN not tracked the costs for ensuring the recovery of the classified supercomputing systems, data, and supporting resources should a servi disruption occur. The classified supercomputing program represents a significant investment, and accountability for these systems is esse Until NNSA clearly defines its component organizations’ roles and responsibilities and fully implements an effective contingency and disaste planning program, it has limited assurance that, in the event of a service disruption, vital information could meet national security priorities. ntial.

Recommendations Executive Action

To improve the effectiveness of contingency and disaster recove planning for NNSA’s classified supercomputing capabilities, we recommend that the Administrator of NNSA direct the weapons laboratories to take the following four actions, where not already implemented: Develop business impact analyses that, among other things, (1) identify and prioritize critical systems, data, and supporting resources; (2) id allowable outage times and impacts for classified supercomputi capabilities; and (3) identify recovery priorities and strategies.

Develop and implement comprehensive contingency and disaster recov plans for all classified supercomputing systems that identify how eac h weapons laboratory’s classified supercom recovered following service disruptions. onduct contingency and disaster recovery plan testing.

Test the three weapons laboratories’ ability to share “on-demand” classified supercomputing capacity to ensure the event of unexpected service disruptions.

In addition, we recomm following five actions: end that the Administrator of NNSA take the Document an agencywide means for reprioritizing the workload across NSA’s classified supercomputing systems should a disruption occur.

Clearly define the oversight responsibilities of the NNSA ASC program office and the NNSA Office of the Chief Information Officer, as they r to contingency and disaster recovery planning for NNSA’s classified s upercomputing operations.

Identify, assess, and communicate the minimum classified supercomputing capacity needed to meet Stockpile Stewardship r equirements in the event of a service disruption.

Develop, document, and implement a process that identifies and tracks expenditures for contingency and d classified supercomputing assets. isaster recovery planning for NNSA’s Develop and document the total anticipated costs for contingency and disaster recovery planning of NNSA’s classified superco which includes the replacement costs for these assets.

Agency Comments and Our Evaluation

In providing written comments (reprinted in app. III) on a draf report, NNSA’s Associate Administrator for Management and Administration agreed that improvements can be made in contingency an disaster recovery planning for supercomputing operations. He indicated that NNSA agreed with six of our nine recommendations and outlined t agency’s intent to conduct business impact analyses, develop and test appropriate contingency and disaster recovery plans, document workload prioritization, and clearly define roles and responsibilities.

However, NNSA did not agree with our recommendation related to identifying the minimum capacity needed to meet Stockpile Stew ardship requirements in the event of a service disruption. The Associate Administrator stated that this recommendation did not take into accou that the different types of supercomputers—capacity and capability— serve different functions and are procured and managed differently. In our report, we recognize that different types of supercomputers exist and that they are used for different purposes, they process unique workloads and operate independently. However, as we point out in the report, althoug h the weapons laboratories have identified supercomputing processing needed for normal business operations, they have not identified the minimum capacity needed to achieve processing priorities in the event of a service disruption. We believe that the recommendation appropriately focuses on meeting NNSA’s Stockpile Stewardship mission and that capacity planning is essential to ensure that information processing and supporting resources exist during contingency operations, regar dless of the type of system used. Although NNSA did not agree with the recommendation, the Associate Administrator stated that the agency will conduct a BIA and build appropriate contingency strategies for both types of supercomputers, as well as enhance capacity siz for contingency and disaster recovery operations.

Further, NNSA did not agree with two recommendations related to identifying and tracking expenditures for contingency and disaster recovery planning and documenting anticipated recovery planning costs, including replacement costs of the assets. The Associate Administrator asserted that this information would not add significant value to managi contingency and disaster recovery planning. However, we believe such actions reflect good government practices and would add value by providing NNSA program managers with useful expenditure and cost information to aid decision making with regards to contingency and disaster recovery planning. As our report points out, GAO’s Standards for Internal Control in the Federal Government states that financial information should be recorded and communicated to program managers to help them make operational decisions and effectively allocate resource for program activities. Strong financial and internal controls are a major part of managing any organization because they help government program managers achieve desired results through effective stewardship of public resources. Accordingly, we believe our recommendations have merit.

We are sending copies of this report to the Secretary of Energy; the Administrator of NNSA; and the Directors of Los Alamos, Livermore, and Sandia laboratories. Copies of the report will also be available to others at no charge on the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact Gene Aloise at (202) 512-3841, or aloisee@gao.gov; Nabajyoti Barkakati at (202) 512-6415 or barkakatin@gao.gov; or Gregory C. Wilshusen at (202) 512-6244 or wilshuseng@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are included in appendix IV.

Appendix I: Objectives, Scope, and Methodology

The objectives of our review were to assess the extent to which (1) the National Nuclear Security Administration (NNSA) has implemented contingency and disaster recovery planning and testing for its classified supercomputing assets, (2) the three laboratories are able to share classified supercomputing capacity for recovery operations, should service disruptions occur, and (3) NNSA tracks the costs for ensuring contingency and disaster recovery planning for its classified supercomputing assets. To address these objectives, we focused on contingency and disaster recovery planning activities at NNSA headquarters, as well as the operating environment for the 12 classified supercomputing systems at the three weapons laboratories—Los Alamos National Laboratory, Livermore National Laboratory, and Sandia National Laboratories.

To assess the extent to which NNSA has implemented contingency and disaster recovery planning and testing for its classified supercomputing assets, we examined contingency and disaster recovery planning controls for the systems within the classified supercomputing environment that are critical to NNSA’s achievement of its nuclear weapons mission. We collected and reviewed policies, procedures, and guidelines from the National Institute of Standards and Technology, the Committee on National Security Systems, the Department of Energy, and NNSA. We also reviewed contingency plans and business impact analyses provided by the weapons laboratories and compared them to federal guidelines. We interviewed NNSA and laboratory officials to determine whether they had documented critical system, data, and supporting resources and whether contingency plans had been tested. Further, we interviewed NNSA officials to determine to what extent they have provided specific guidance and oversight for the laboratories to ensure that contingency and disaster recovery planning requirements are being met.

To determine the extent to which the three weapons laboratories have the ability to share supercomputing capacity for backup and recovery operations, we visited each weapons laboratory and gained an understanding of the overall classified supercomputing infrastructure and identified interconnectivity and control points. We performed technical assessments of supercomputing capabilities at each weapons laboratory, including each laboratory’s ability to share supercomputing capacity under normal operating conditions. We reviewed the weapons laboratories’ efforts to determine the minimal supercomputing capacity needed to meet NNSA Stockpile Stewardship Program requirements along with the ability of the weapons laboratories to share supercomputing capacity on an “on- demand” basis, including the use of advanced architecture systems. In addition, we obtained documents describing the supercomputing system environment as well as capacity information, along with the views of officials from NNSA and the three weapons laboratories.

To assess the extent to which NNSA tracks costs for ensuring contingency and disaster recovery planning for classified supercomputing assets, we interviewed NNSA and weapons laboratory officials to determine how expenditures were tracked for contingency and disaster recovery planning of the classified supercomputing systems at each of the laboratories. We also requested the amount of funds NNSA obligated to the three weapons laboratories, and the amount of funds the laboratories spent, in implementing NNSA’s classified supercomputing capabilities from fiscal years 2007 through 2009. Further, we interviewed NNSA and laboratory officials to determine how they projected future cost estimates for ensuring the recovery of these assets for fiscal years 2011 through 2014. To assess the reliability of data provided, we reviewed (1) the fiscal year 2009 financial statement audit for the system and (2) responses NNSA provided to questions about processes and procedures for ensuring the accuracy and completeness of data. Based on this information, we determined the data are sufficiently reliable for the purposes of this report.

We conducted this performance audit from December 2009 through December 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: NNSA Annual Obligations for Its Advanced Simulation and Computing Program, Fiscal Years 2007 through 2009

NNSA reported obligating approximately $1.7 billion from fiscal years 2007 through 2009 to support Advanced Simulation and Computing (ASC) program activities at the three weapons laboratories. The $1.7 billion was used mainly for three efforts: Weapons codes and models. This effort is intended to develop and improve weapons simulation codes and models for predicting the behavior of weapons systems and devices in the nuclear stockpile.

Computational systems and software environment. This effort is intended to provide ASC users a stable, seamless computing environment for ASC-deployed platforms. It is responsible for procuring, delivering, and deploying ASC computational systems and user environments via technology development and integration across the three weapons laboratories.

Facility operations and user support. This effort is intended to provide both the necessary physical facility and operational support for reliable supercomputing and storage environments, as well as a suite of user services for effective use of the three weapons laboratories’ computing resources. Facility operations cover physical space, power and other utility infrastructure, and local- and wide-area networking, as well as system administration, cyber security, and operations services for ongoing support. The user support function includes planning, development, integration and deployment, continuing product support, and quality and reliability activity collaborations.

Figure 4 depicts NNSA’s annual obligations for each of the three efforts from fiscal years 2007 through 2009.

As shown in figure 4, NNSA annual obligations for its classified supercomputing operations decreased from about $591 million to $529 million between fiscal years 2007 and 2009. The largest obligation for the classified supercomputing program was for weapons codes and models, which accounted for approximately $750 million (or 45 percent) of total obligations.

Obligations for computational systems and software environment accounted for approximately $527 million (or 32 percent) of total obligations. For the period, obligations for this effort decreased from $183 million to $161 million.

The facility operations and user support activities, which includes, among other things, expenditures for contingency and disaster recovery planning, accounted for $390 million (or 23 percent) of total obligations over the period. These obligations ranged from $114 million to $147 million for the 3 fiscal years.

Appendix III: Comments from the National Nuclear Security Administration

Appendix IV: GAO Contacts and Staff Acknowledgments

Staff Acknowledgments

In addition to the individuals named above, Glen Levis, Edward M. Glagola, Jr., and Jeffrey Knott (Assistant Directors); and Preston S. Heard, Jennifer R. Franks, Kevin Metcalfe, and Zsaroq Powe were key contributors to this report. Neil Doherty, Nancy Glover, Franklin Jackson, and Jonathan Kucskar also made key contributions to this report.