# Presentation: 757638

## SELECTING A DISASTER RECOVERY SITE

- D. J. BRAGG
- SANDIA NATIONAL LABORATORIES

- Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000.

## BACKGROUND INFORMATION

- Preliminary study in 2005 to characterize SNL’s ability to recover from interruption of data center operation
- Limited off-site storage due to security requirements
- Designated recovery site ~5 miles from main data center
- Data center loss would require > 6 months to recover ~80% of applications and data

## BACKGROUND INFORMATION

- 2006 study to find and designate a disaster recovery site
- Evaluation team included SNL staff and Disaster Recovery/Business Continuity consulting firm
- Study included government owned, commercial, and build-to-suit facilities
- Selection criteria established with weighted grading
- Criteria included environmental, infrastructure, geologic, social, financial, regulatory and political considerations.
***The obvious answer is not always the correct answer***

## WHAT ARE THE OPTIONS?

- Outsourcing to a commercial or non-SNL facility
- Pro
- Equipment on hand and fully maintained
- Ability to leverage quick ship agreements with vendors
- 24x7 operations, physical security
- Con
- Limited access and  availability for testing
- Data not under your direct control
- Require long-term, expensive contract
- Insourcing to user controlled facility
- Pro
- Lower cost
- Greater chance of meeting RTO
- No scheduling issues for testing/availability
- Control and security over data
- Con
- Facility identification, acquisition and maintenance
- Underutilized equipment, equipment maintenance

## Tier 4: Multiple active power and cooling distribution paths, redundant components, fault tolerant, 99.995% availability
Tier 3: Multiple power and cooling distribution paths, but only one path active, redundant components, concurrently maintainable, 99.982% availability
Tier 2: Single or multi path for power, single cooling distribution path, redundant components, 99.741% availability
Tier 1: Single path for power and cooling distribution, no redundant components, 99.671% availability
Industry Standard Tier Classifications – The Uptime Institute

_**Tier 3**_: Multiple power and cooling distribution paths, but only one path active, redundant components, concurrently maintainable, 99.982% availability
_**Tier 2**_: Single or multi path for power, single cooling distribution path, redundant components, 99.741% availability
_**Tier 1**_: Single path for power and cooling distribution, no redundant components, 99.671% availability
*Industry Standard Tier Classifications* – **The Uptime Institute**

**INDUSTRY STANDARDS**

|  | Terminology |  | Definition |
| --- | --- | --- | --- |
|  | 10 | State-of-the-Art | Redundant power, redundant cooling, redundant UPS, redundant dedicated A/C, redundant generator, redundant fuel, weather & geographic facility hardening, disaster avoidance |
|  | 9 | Ultra-Reliable | Redundant power, redundant cooling, redundant UPS, redundant dedicated A/C, redundant generator, redundant fuel |
|  | 8 | Reliable-Redundant | Dedicated power & cooling, redundant UPS, redundant dedicated A/C, redundant generators |
|  | 7 | Reliable | Dedicated power & cooling, UPS, redundant dedicated A/C, generator |
|  | 6 | Isolated Mostly Reliable | Dedicated power & cooling, UPS, redundant dedicated A/C |
|  | 5 | Isolated Improved | Dedicated power & cooling, UPS, dedicated A/C |
|  | 4 | Isolated Conditioned | Dedicated power & cooling, conditioned power, dedicated A/C |
|  | 3 | Isolated Unreliable | Dedicated power & cooling, unconditioned power, dedicated A/C |
|  | 2 | Partially Isolated Unreliable | Dedicated power, shared cooling, unconditioned power, A/C |
|  | 1 | Unreliable | Shared building power & cooling |

**1**

**2**

**3**

**4**

## SELECTION PROCESS

## Slide 7

| CRITERIA | DESCRIPTION | RATING |
| --- | --- | --- |
| SITE LOCATION CRITERIA |  |  |
| Site Location Specification | Downtown/city center, office/high tech park, suburban, industrial park, parking, shipping access, etc.  | A |
| Access to Facility | Remoteness/location of the facility.  Requires more than one access road  | A |
| Environmental Disaster Avoidance | Requirements for the facility that it not be near earthquake/fault lines, tornado, not in 100 year flood plain, mudslide or rockslide area  | B |
| Distance from 880
(Data Center) | Not less than 50 Miles and up to 800 miles away.  Tradeoff between communication latency issues, accessibility, and survivability.  | B |
| Market Location | Location of Recovery Center in a Tier I/II/III city.  May impact cost and infrastructure considerations  | B |
| Geography Rank | Location for the facility within the United States.  | C |
| SECURITY CRITERIA |  |  |
| Rights of Access | Provisions for DOE complete control of access to facility.  | A |
| Classified Processing | Provisions to meet DOE requirements for processing classified information.  | A |
| Physical control of facility | Physical control of facility for security reasons and immediate access.  | B |

## Slide 8

| CRITERIA | DESCRIPTION | RATING |
| --- | --- | --- |
| FACILITY CRITERIA |  |  |
| Tier 3 Facility | Tier 3 - Multiple power and cooling distribution paths, with only one path active, redundant components, concurrently maintainable, 99.98% availability.  (DR Study Phase 1 requirement) | A |
| Infrastructure | Electrical and telecommunications feeds, floor loading, raised floor height, available raised floor.  | A |
| General Building Specifications | Building Height, Class, Age, etc.  | A |
| Fire Suppression | FM-200 Fire Suppression System. DR Study Phase 1 Requirement  | B |
| Additional Conditioned Raised Floor | Additional raised floor to stage equipment on conditioned raised floor and area to support immediate growth.  | B |
| Primary Building Use | Primary use of building, i.e. laboratory, manufacturing, data center, recovery center, office, mixed use, other  | B |

## Slide 9

| CRITERIA | DESCRIPTION | RATING |
| --- | --- | --- |
| USAGE CRITERIA |  |  |
| Costs | Site cost, labor pool availability, proximity to 880, infrastructure, connectivity, etc.  | A |
| Length of Usage | Potential for restrictive time limits for use if using a commercial provider.  | A |
| Infrastructure Disaster Avoidance | Away from Airport, Highways, railroad tracks, electrical sub-stations.  | A |
| Political Considerations | Considerations based on external political factors  | B |
| Ownership | Sandia leased or owned, DOE leased or owned, military leased or owned and service provider leased or owned, lease expiration dates.  | B |
| Accommodations for Support Staff | Availability of hotels and long-term accommodations to house support staff potentially for extended periods of time.  | B |
| Food Catering Services | Availability of balanced meals should be available for an extended outage.  | B |

## MOTHER NATURE’S RECORD

## OTHER CONSIDERATIONS

- What is the likelihood of a disaster occurring at both the primary site and the recovery site?
- What potential actions or conditions could prevent access to the recovery site?
- Military installation closure
- Civil disturbances, protesters
- All flights grounded (9/11)
- What about other “environmental” aspects
- Proximity to interstate (fuel/chemical truck accident)
- Proximity to fuel storage facilities, industrial operations
- Seasonal blizzards, poor weather making travel hazardous

## WHAT COULD POSSIBLY HAPPEN HERE?

## THORNY ISSUES

**Physical and Data Security**
- Security for facility, access control
- VTR Requirements for classified
- What about Sigma data?
- Clearance and badging of facility staff
- Surveillance, monitoring and auditability
**Adequate funding for startup and continued operation**
- New facility ~$10-15M
- Update existing $1-5M
**Foreseeable technology changes**
- Critical for planning of cooling, electrical
- 30kW racks in very new future
- 10G fibre

## SCREENING PROCESS

- Survey questionnaire with 165 items addressing all criteria sent to candidate facilities
- Returned questionnaires were scored and ranked
- Site visits were conducted to qualitatively assess the site based on facility condition, existing operations, potential for growth, etc.

## SITES CONSIDERED

- NWC Facilities
- LANL (10), KCP, LBL, LLNL, SNL (2), NNSA Abq Service Center
- Commercial
- Qwest (Albuquerque), Big Byte (Albuquerque), Qwest Cyber Center (Denver), Recovery Point Systems (Gaithersburg), Qwest Cyber Center (Sterling VA), Qwest Cyber Center Sunnyvale, Lockheed Martin Information Technology Data Center (Beltsville MD), Lockheed Martin Enterprise Information Systems (Orlando FL), Verizon Data Services Inc. (Temple Terrace FL), Verizon (NOC) DFW Airport, Verizon Enterprise Solutions (Arlington VA), VeriCenter Denver, VeriCenter Atlanta, Matic Site Laguna Industries (Laguna Pueblo)

## FINAL FOUR

- Commercial Recovery Site
- Vericenter, Denver
- Established commercial operation
- Close to Albuquerque
- Build-To-Suit
- Matic Facility, Laguna Pueblo
- 50 miles from SNL
- Long term lease agreement needed
- DOE/NNSA
- Honeywell (FM&T), Kansas City
- Would require line item authorization
- Sandia Livermore
- Sandia controlled
- Would require some facility upgrades

## TACTICAL VS. STRATEGIC

**Tactical approach addresses immediate needs with existing resources and timeline**
- SNL California selected as best fit
- Under SNL control
- Existing facility and trained staff
- Require some facility upgrade and modification
- Located in severe earthquake zone
**Strategic approach for long term capability**
- Establish a third facility for both NM and CA
- Build a dedicated facility at another location
- May be government owned or long term lease/option
- Potential for “shared” facility, recovery capability with other NWC sites

- QUESTIONS?