# Presentation: 562912

## DESIGNING A DISASTER RECOVERY STRATEGYSTEP 1: APPLICATION TIERING

- D. J. BRAGG
- SANDIA NATIONAL LABORATORIES

- Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000.

**Notes:**
Recovery of data and IT resources is highly complex and involved task that requires a lot of work.  It is very easy to say “we’ll just get new systems and recover from tape.”  Realistically, you have to take into consideration what needs to come back first and what can wait till later.

## THE PROBLEM

- Your Enterprise compute systems, including infrastructure and business applications, are housed in a single data center
- You need to develop a recovery strategy to recover over 200 applications AND all necessary infrastructure components in the event of a significant outage or loss of the site itself
- So what will you recover first?

**Notes:**
Does the number 200 applications seem large?  
Many companies seem to survive with just five or ten applications to do business...Oracle or SAP have integrated CRM and business intelligence suites that do it all.  Of course you have email and office software suites, but for the most part, the desktops are locked down and there is no chance of just adding a new application.  
We have a much different problem.  We deploy specialized applications, modeling tools, you name it, to do our diverse and complex tasks.  That adds to the enterprise overhead, and results in many discrete systems on our networks.  Two hundred is not  a large number when you look at what we do.

- IT RECOVERY STRATEGYA Systematic Approach

- Replicate critical data assets
- Initiate recovery of mission critical applications within an agreed upon time objective
- Essential IT services
**Tiering** **process** to identify mission critical applications
- Business Impact Analysis (BIA)
- Expand IT operations to near normal state
- Continued operation of alternate facility for as long as necessary until return to primary facility
- 1 month, 1 year?

**Notes:**
We needed to come up with a logical approach to recovering our IT environment in the event of a disaster.  You can’t do everything at once so you have to take a measured approach to recovery.
When we looked at what we had in our enterprise, it was obvious that the “data” was our most important asset and that the first priority was to protect that data from loss due to some disaster scenario.  The solution here is easy – replicate the data to some place where it is safe.
The next step in our approach is to classify the services and applications based on their importance of returning Sandia’s IT environment to near normal condition.  
The critical IT services are the first priority.  You have to have the capability of logging into a network to access the applications and data.  So DNS, Active Directory, authentication, etc. are the first to reappear.  But what comes next?
We used a TIERING method to make that determination and the BIA is the tool necessary to do that.
The BIA will evaluate each business function and determine what resources are needed to return to normal operating conditions.
In our case, our approach is to set up a recovery site for our most important application and services, expand those operations to a near normal state, and allow of continued operations for a long as it takes until we can return to our primary facility – whether that takes a month or a year or longer.

- APPLICATION TIERING

- Tiering is a relative assessment to determine the priority and necessity of a service or function
- What is needed to do business?
- What criteria must be met?
- Can the function or service be provided by some alternative method?
- How long can we live without the service or function?

**Notes:**
Tiering is not a clear-cut process and requires close partnerships with the business units outside of IT.
There are strict rules on WHAT business application HAVE to happen, specifically pay your people and pay your suppliers.  Beyond that, you have to take a look at what else is needed for your company to survive.
 What IT functions or applications are required keep the business alive?
What criteria are you using to determine the criticality of an application or service?
For example:  Do you have to have email up and running or do you have alternate methods for communicating with your staff?
How about an external web pressence?  Do you need that to communicated or do business?
Can you use paper forms instead of the web application?  Do you even have the paper equivalent available?
Most critical, how long can you “live” without the application or service before it seriously affects our mission?

- DETERMINING CRITERIA

- Tiering should agree with the predetermined recovery objectives
- RECOVERY **TIME** OBJECTIVE **(RTO)**
- The time needed to bring the application or service back on line at time of declaration (ATOD)
- Time to regain function, service, etc.
- RECOVERY **POINT** OBJECTIVE **(RPO)**
- The allowable gap between when the data or service was last updated.
- Degree of data loss

**Notes:**
Two principles affected how we prioritized the services and applications.  
The RTO is a basic guideline – a gut feeling – on how long it could take to bring the application and service on  line and available on the network.  In some cases, the RTO is a negotiation between the application or service owner and the recovery team.  If you have to have access to the network, meaning ability to log on and authenticate, it could be  a short RTO.  However, if you can wait four or eight hours (or for that matter eight days) for a service to resume, then the RTO should reflect that.

The RPO is a measure on how long a data set or service can go without being updated to be usable.  For example, we update our DNS approximately every 15 minutes from a master network information database.  If that database was unavailable for a few hours, that may only affect a small number of addresses.  In a financial institution, data is very dynamic and changes by the second.  A period of eight hours could spell the difference between life and death to that institution.

- WHERE  WE WERE

- Recovery capability > 36 weeks
- Not all data was backed-up offsite
- Critical databases, application programs
- Relied on repurposing development systems
- Strategy required ‘local’ critical application recovery
- Uncharacterized environment
- No maintained server, application inventories
- Infrastructure services not redundant
- Unknown relationships between IT services, applications, and mission functions

**Notes:**
Two and half years ago, we conducted a study to determine how long it would take to bring up our data center in the event of a catastrophic incident that rendered the center unusable.  It was found that it would take more than 36 weeks to recover approximately 90-95% of the current applications and services hosted.  

We found that a good amount of our data was backed up locally, and only a portion of the information was off-sited.  Our recovery strategy centered on repurposing development and quality systems to production, but we did not have enough equipment to support more than a couple of applications.

The biggest surprise was how complex our environment had become.  The mainframe days were over, and client-server now reigned.  We had allowed our environment to grow without really paying attention to the complexity that was being built into it.  There were many uncharted cross dependencies and not all of them were documented.

In order to “eat the elephant” we came up with the tiers that would allow one bite at a time.

- IT APPLICATION AND SERVICE TIERS

- TIER 0 – *Survive*
- The necessary services and/or applications to protect life, property, or basic communication
- TIER 1 – *Exist*
- The applications and services allowing Enterprise to function at a minimum capability, provide basic communications, meet regulatory requirements
- TIER 2 – *Operate*
- The applications and services that allow *almost* normal operations, though at a degraded or metered state
- TIER 3 – *Enhance*
- Less critical applications and services.  The ability to create new work, update current services, add new capability

**Notes:**
At this point in our discovery process, the DR team felt very much like Tom Hanks in the movie “Cast Away”.  We were up to our ears in information, we had a clear goal, but we were mired down by the sheer numbers of applications and services that had to be recovered – at that time 250+ and growing.  We set up a four tier system, and assigned each a “comfort” rating.  

The lowest being survival – just breathing and barely alive.  No need to get fancy here.

The next level was bare existence.  Ability to keep ourselves fed, with minimum creature comfort.

The third level was approaching near normal.  We could recover most of the services and applications needed to do business, but not necessarily in the data center or anyplace of permanance.

The final level is the ability to be back to normal in our new or restored home.  We can now develop new software, add new services, and basically back to where we were before the disaster hit.

## SURVIVAL MODE – TIER 0

- Physical security systems, Emergency Support, Fire detection and alarm
- Emergency operations function – not IT
- Evaluate what is necessary to provide the _minimum_ connectivity
- Core network operations
- Authentication and authorization processes
- Active Directory
- Kerberos, 2 Factor Authentication
- Minimum external web presence to communicate to staff, suppliers, customers
- Data verification (backups, mirror, etc)
- RTO – 0 to 2 hours

## EXIST MODE – TIER 1

- The applications necessary to keep the business alive
- Mandated by regulatory requirements
- Payroll and Pay Suppliers
- Communications to employees
- Email, IM, minimum internal web services
- Mission critical services
- Nuclear Emergency Search Team (NEST)
- Accident Response Group (ARG)
- Data recovery
- RTO – 24 to 48 hours
- Dedicated systems to provide functionality
- May require scheduling of specific application “windows” of operation

**Notes:**
Tier 1 –
This tier addresses the services and applications that establishes and maintains a minimal business presence.
We have two main applications that have to work – payroll and accounts payable.  
We also want the ability to establish internal communications to the staff and our external suppliers, so email is a tier one function.
There are some non-IT applications and services that are critical to Sandia’s mission, such as supporting NEST and ARG
This is the point where we begin to recover our data from the offsite storage and make it available to applications and end users.  
The relatively short RTO – 1 to 2 days – a real need for these services.  We have equipment on hand to run these applications, however, it will be assumed that we will have set up schedules on when certain apps, such as Oracle, will be online, and the resources will be dedicated to other services as necessary.

## OPERATE MODE – TIER 2

- The applications and services to provide support functions
- Mission support applications
- Specialized tools, graphic apps, user and data collaboration
- Training, Foreign National Request (FNR)
- Business support functions
- Financial reporting, supply chain, badging
- RTO – 5 to 30 days
- Additional equipment procured by quick ship, JIT procurement

**Notes:**
Tier 2 –
At this point, the enterprise should begin to resume some degree of normality.  This doesn’t necessarily mean that the data center is open for business, or that all applications are being run from the recovery site.  Depending on the type of disaster, the data center may have limited functionality (or an alternate temporary data center is in operation), but more applications are coming back online.  
If there was major destruction of systems, then we will utilize JIT or quick ship contracts to get replacement systems and equipment.
Tier 2 brings more services and applications to the user, but again, all applications will not be running all the time.  

## ENHANCE MODE – TIER 3

- Less critical business support applications
- Travel, benefits
- Department level support applications
- New application development and deployment
- May be hosted at recovery site or  a local temporary facility
- RTO – 30 to 60 days
- Will require additional equipment not through quick ship agreement or JIT will require longer delivery time

**Notes:**
Tier 3 –
This tier is the point where things are returning to normal.  
The “creature comfort” applications are coming online, and we can begin to upgrade or add new apps to the environment.  
This phase is not necessarily a return to the data center.  If the building were destroyed, it may take up to a year to replace, but for the most part, operations are “normal” and routine activities are taking place.

## Slide 12

| Tier | Application / Service | IT DR Service Level and Strategy | Examples |
| --- | --- | --- | --- |
| Tier 0
Survive
Access,  Safety, Life Support |  Infrastructure Services to provide user access and authentication, minimal communication
  Heat, Light, Water
  Life Safety Systems |   IT RTO – 0-2 hours     
  Immediate need        
  Dedicated recovery systems, near automatic fail-over |          Emergency Response
         Sprinkler systems, fire alarms
         Physical security
         Network access and authentication
         www.sandia.gov |
| Tier 1
Exist
Essential mission operation
  |   Business critical systems mandated by regulatory requirements 
  Communication processes
  Applications and services necessary for support of mission critical functions |   IT RTO 24-48 hours
  Dedicated recovery systems, manual execution of processes |          Payroll application
         Accounts payable (Oracle Fin)
         NWIS (SNL Network Application)
         Email (dial-tone)
         NEST, ARG
         Infographics
         Minimal web presence |
| Tier 2
Operate
New requests (adds/changes) |   Mission support applications and services
  Business support applications and services |   IT RTO 5-30 days
   Systems will be procured At Time of Disaster Declaration (ATODD) from specific vendors |          Supply chain (Procurement)
         Training applications (TEDS)
         Foreign National Requests (FNR)
         Badgeing Application
         Financial Reporting |
| Tier 3
Enhance
Develop  |   Less critical business applications |   IT RTO 30-60 days
  Systems will be procured as planning evolves   |          Departmental support applications
         Travel
         Benefits |

## STRATEGY RATIONALE

- All services and applications cannot be brought back on line at the same time.
- Bring up what you have to when you need to
- Avoid repurposing of D, Q, P systems
- Core services redundant (Active Directory, DNS,Kerberos)
- Significant cost in maintaining 75% to 100% capability at designated recovery site
- Facility and Equipment
- Manpower to maintain systems in current levels
- Use existing technology and similar implementation for all systems
- Disaster Recovery NOT an opportunity to adopt new, bleeding edge technologies

**Notes:**
When you have over 300 applications and services running on your enterprise networks, you cannot realistically expect to bring everything back at once or even over a few months.  
9/11 shook the IT world.  Many businesses located in the NY area were impacted, and it took months to restore even basic functionality to those most affected.  Those businesses that had rigorous DR plans survived.  This that didn’t don’t exist anymore.
Some of the financial institutions were running parallel environments, making DR easy.   Others had established offsite recovery operations and were back online in a few days or weeks.
Realistically, we can’t have parallel centers, ready to take over immediately.  It would cost millions to establish and maintain that capability.
Another issue that is hard to address is that we all are currently running very diverse and complex environments.  The manpower neccessary to maintain those environments is a very specialized and highly trained population.  To recover the talents and knowledge of those people is very difficult, thus replacing them will require unique and extraordinary measures.  Documenting your systems and processes is essential.  
A recovery operation is not an opportunity to try out new things.  You won’t have the luxury or time to adopt new technologies.  You have to get going again as fast as possible.

## THE CHALLENGE

- You have to identify what you have, what are the interdepencies, and what you _***really have to have***_ to support your enterprise
- Building a strong working relationship with your customers to know what they need and when
- Business Impact Analysis of all operations completed prior to time of disaster by business units with guidance from DR/BC team
- Periodic testing of DR environment to build confidence with business units
- Periodic review of applications tiering relationships to determine any changes of priority or support requirements

**Notes:**
So here’s a challenge for you today.,
 How well do you know your environment?
 How complex is it?  Do you know how your applications and services really work and all the interdependencies between the applications and data?  
 How much data do you have to protect and how fast can you get it back?
 Have you done Business Impact Analyses with your customers, business units, and suppliers?
 When was the last time you tested one of the Tier 0 applications, such as DNS or your Active Directory?
 Have any processes, procedures, or applications changed over the last 12 months and is all the necessary knowledge captured that will allow you to recover? 

## THE CHALLENGE

- Knowing when to say “no” and realizing the real limitations of a disaster recovery effort
- Prepare for all business functions expecting or wanting to recovery early in the tiers
- Be ready to negotiate and be open to any external requirements/mission
- Accurate and up to date documentation of systems and applications
- Small application or system changes can kill a recovery operation

**Notes:**
And don’t assume that everything you have running now will be able to be recovered.  
Disaster recovery is just that.  Recovery from some bad thing under adverse conditions and a lot of yelling going on.
Your business units have to know your limitations, and you have know their basic mission needs.
You will have to negotiate to a great degree, and cooperations is essential.
Again, documentation is critical, and there are a lot of pitfalls that can prevent recovery.
All you can do is try to find all the traps before your back is to the wall.

## CONCLUSIONS

- Application tiering is a systematic approach to determine the order in which applications and services are brought back on line in a measured and planned operation
- Planning is only the first step – testing is required to ensure service failover and recovery of data and services

**Notes:**
Tiering is critical to organizing a recovery effort.  

Three critical elements are
 Plan
 Test
 Retest

- QUESTIONS?