Appendix J -- Guidelines for Application Hot Site Tests

# Purpose and Scope

The purpose of this guideline is to provide general instructions to the
user community who will be testing their applications at a hot site test
(HST). This guideline presents:

-   General HST procedures;

-   The schedule for the next hot site test;

-   General advice on preparing for the HST; and

-   Platform specific information for Titan and EOS, including:

    -   A brief description of the hot site test environment,

    -   Specific instructions for ensuring critical files will be
        available for the HST,

    -   Cautions on potential problem areas with solutions (to ensure
        the application tests will avoid common mistakes in order to
        concentrate on truly testing application disaster preparedness),
        and

    -   Procedures to follow on the day of the test.

General information is presented first, followed by two separate
subsections covering platform specific information for Titan and EOS
testing, respectively.

# General Hot Site Test Procedures

The hot site test is scheduled for a three day period with:

-   Days one and two dedicated to testing the recovery of the operating
    environments and recovering the application software and data by the
    DR support team, and

-   Day two being dedicated to customer testing of their recovery
    operations.

Prior to the HST, the DCSS Disaster Recovery staff determines which
backup tapes will be used for the tests. This establishes the *Recovery
Point-in-Time*. The Recovery Point-in-Time represents the state of the
systems as they existed at the time the backup tapes were made. This
Recovery Point-in-Time will lie somewhere between one to two weeks prior
to the date of the actual test. All user preparation, including
placement of datasets on volumes that are restored at the hot site, must
be completed [BEFORE]{.underline} this date.

During the tests, in addition to providing customer assistance, the DR
staff maintains a running log of the test activities to provide current
test status to customers participating in the DR testing and to assist
in the post-test reviews.

After every test, debriefings are scheduled to review both the testing
of the system and application recovery procedures.

# Note Regarding Hot Site System Performance

Be advised that we cannot exactly duplicate production configurations at
the disaster recovery hot site. We are limited by the available
selection of equipment from the hot site vendor as well as cost
considerations. DCSS makes every effort to obtain hot site
configurations that best support customer disaster recovery needs, but
performance will not always match that of production. If you experience
extremely poor performance during DR testing, please contact us at the
time you have the problem, so that we might be able to ascertain where
the problem is. This will help us with planning future modifications to
our disaster recovery configurations to provide more effective service.

# July 15, 2008 Hot Site Test Schedule

**In order to ensure all application files are backed up on the selected
Recovery-Point-in-Time, be sure the application files are stored on the
proper volumes by close-of-business on the following dates:**

> **Friday, July 4, 2008 -- Titan System**
>
> **Friday, June 27, 2008 - EOS**

+--------------------+-------------------------------------------------+
| ***Sunday July 13  | ***System Recovery and Communications Testing   |
| -***               | (CIT Staff only)***                             |
|                    |                                                 |
| > ***Monday July   |                                                 |
| > 14***            |                                                 |
+--------------------+-------------------------------------------------+
|                    |                                                 |
+--------------------+-------------------------------------------------+
| 9:00 am Sunday --  | On Sunday, DR staff will start restoring all    |
|                    | Unix operating systems, applications, and       |
| > 8:30 pm Monday   | databases. On Monday, DR staff will complete    |
|                    | the Unix restores, restore all z/OS disks and   |
|                    | IPL the system, and establish and test the data |
|                    | communications links in preparation for         |
|                    | application testing.                            |
|                    |                                                 |
|                    | As the Unix systems restorations are completed, |
|                    | they will be turned over to customers to        |
|                    | complete their restoration procedures.          |
|                    |                                                 |
|                    | Customers do not officially test the            |
|                    | applications until the designated time, but     |
|                    | based on past test experience, the systems may  |
|                    | be up and running earlier. Users are welcome to |
|                    | test their connectivity after the DR staff has  |
|                    | announced that you may do so. The systems will  |
|                    | remain up overnight, and users may perform      |
|                    | testing if desired, but CIT staff will not be   |
|                    | available to resolve any problems until the     |
|                    | official customer test time.                    |
+--------------------+-------------------------------------------------+
|                    | #########                                       |
+--------------------+-------------------------------------------------+
| ***Tuesday July    | ***Application Testing***                       |
| 15***              |                                                 |
+--------------------+-------------------------------------------------+
|                    |                                                 |
+--------------------+-------------------------------------------------+
| 8:00 am -- 4:00 pm | Titan and EOS DR systems are available for      |
|                    | testing applications.                           |
+--------------------+-------------------------------------------------+
|                    |                                                 |
+--------------------+-------------------------------------------------+
| 4:00 pm - 1:00 am  | Customers may continue testing, but be advised  |
|                    | there will be no full CIT staff support to help |
|                    | resolve problems.                               |
+--------------------+-------------------------------------------------+

# Test Support Contact Information, Tuesday July15, 8:00 am - 4:00 pm

***Titan Support:*** **(301) 496-5181**

**Provide your name and contact phone number and state you are calling
for "Disaster Recovery Support"**. We will take your information and
have the appropriate support person contact you within five minutes.

***Unix Support:*** **Submit an EXPEDITED ASR that clearly states it is
a DR test request.**

Alternately, you may call (301) 496-9160. Provide your name and contact
phone number, identify which DR system you are using (e.g., EOS, NBS,
etc.) and state you are calling for "Disaster Recovery Support". We will
take your information and have the appropriate support person contact
you within five minutes.

# Web site information: dr.cit.nih.gov 

On the days of the test, a Web site will be available with a posting of
the status of the recovery process and other useful information.
Customers can check this Web site for updates indicating hot site
systems' availability.

# General Contact Information

Please contact Adrienne Yang, Disaster Recovery Coordinator, at (301)
496-1053 with any questions or concerns.

# Next Scheduled Test Date: Tuesday, December 2, 2008 

# Preparing for the Hot Site Test

*Ensure All Critical Files will be Available* -- Maintain a
**[complete]{.underline}** list of **[all]{.underline}** files required
for processing the application in recovery mode after a disaster. If the
application uses cataloged procedures or Job Control Language on the
mainframe, or *cron* jobs on EOS, review each of them to identify all
files needed to successfully execute the procedures/jobs. Also, review
all command procedures/script files that may reference files that are
created, used, or updated. Reference the platform specific sections for
further instructions for ensuring the availability of critical files for
testing.

*Use the Password that was in Use Prior to the Recovery Point-in-Time
Date for the System* -- All system and user passwords reflect the
password in use at the point-in-time the system backup tapes were made.
Therefore, users who will be participating in the application HST must
be sure to use the password that was in effect prior to the date of the
[system]{.underline} backup (listed above). [This is particularly
critical for those users who have changed their passwords between the
time of the system backup and the date of the HST.]{.underline}

*Ensure Firewalls are Configured to Allow Communication with the Hot
Site* -- Update firewalls with the following disaster recovery IP
addresses/hostnames, as appropriate:

+-----------+--------------+-----------+------------+-----------------+
| **        | ***Disaster  | ***P      | ***Pro     | ***CIT          |
| *Disaster | Recovery***  | roduction | duction*** | Enterprise***   |
| Re        |              | IP        |            |                 |
| covery*** | **           | A         | ***H       | ***User         |
|           | *Hostname*** | ddress*** | ostname*** | Services***     |
| ***IP     |              |           |            |                 |
| A         |              |           |            |                 |
| ddress*** |              |           |            |                 |
+===========+==============+===========+============+=================+
| 165.11    | drti         | 128.      | tita       |                 |
| 2.213.228 | tan2.nih.gov | 321.64.66 | n2.nih.gov |                 |
+-----------+--------------+-----------+------------+-----------------+
| 165.11    | drt          | 128.      | tit        |                 |
| 2.213.229 | itan.nih.gov | 231.64.34 | an.nih.gov |                 |
+-----------+--------------+-----------+------------+-----------------+
| \"        | drcd.t       | \"        | cd.tit     | Connect:Direct  |
|           | itan.nih.gov |           | an.nih.gov |                 |
+-----------+--------------+-----------+------------+-----------------+
| \"        | drftp.t      | \"        | ftp.tit    | FTP             |
|           | itan.nih.gov |           | an.nih.gov |                 |
+-----------+--------------+-----------+------------+-----------------+
| 165.11    | drcorvus     |           |            | eRA             |
| 2.213.238 | .cit.nih.gov |           |            |                 |
+-----------+--------------+-----------+------------+-----------------+
| 165.11    | dreos        | Multiple  |            | UNIX Alpha      |
| 2.213.240 | .cit.nih.gov | addresses |            | Servers         |
+-----------+--------------+-----------+------------+-----------------+
| 165.11    | drweb1       | Multiple  |            | DR Web host     |
| 2.213.242 | .cit.nih.gov | addresses |            |                 |
+-----------+--------------+-----------+------------+-----------------+
| 128.23    | drlapis      | Multiple  |            | PMS             |
| 1.164.003 | .cit.nih.gov | addresses |            |                 |
+-----------+--------------+-----------+------------+-----------------+
| 128.23    | drgarnet     |           |            | PMS             |
| 1.164.011 | .cit.nih.gov |           |            |                 |
+-----------+--------------+-----------+------------+-----------------+
| To be     | To be        |           |            | NBS             |
| d         | determined   |           |            |                 |
| etermined |              |           |            |                 |
+-----------+--------------+-----------+------------+-----------------+
| 128.2     | drracer      |           |            | EHRP            |
| 31.164.35 | .cit.nih.gov |           |            |                 |
+-----------+--------------+-----------+------------+-----------------+

*Ensure Connect:Direct Procedures Use the Correct snode* -- If you use
the *host name* to specify the system (snode=\<*hostname*\>) to which
you are connecting, you will have to change it for the hot site test
before you can run C:D. The following table lists the DR host names to
use.

+-----------------------------------+-----------------------------------+
| ## Disaster Recovery Host Name    | ***Production Host Name***        |
+===================================+===================================+
| drcd.titan.nih.gov                | cd.titan.nih.gov                  |
+-----------------------------------+-----------------------------------+
| dreos.cit.nih.gov                 | eos.cit.nih.gov                   |
+-----------------------------------+-----------------------------------+
| drgarnet.cit.nih.gov              | garnet.cit.nih.gov                |
+-----------------------------------+-----------------------------------+

If you use the *node name* to specify the system to which you are
connecting you don't have to make any changes.

We highly recommend that if you are using host name, you change your
procedures to specify node name. The following table lists the node
names:

+-----------------------------------+-----------------------------------+
| ## System                         | ## Node Name                      |
+-----------------------------------+-----------------------------------+
| Titan                             | nih.std.ndm                       |
+-----------------------------------+-----------------------------------+
| eos                               | eos.ndm                           |
+-----------------------------------+-----------------------------------+
| garnet                            | garnet.ndm                        |
+-----------------------------------+-----------------------------------+

The following is an example of a Connect:Direct process (for UNIX) using
the node name to specify the Connect:Direct node to which you are
connecting:

> tocu process snode=nih.std.ndm snodeid=(uuu,ppppppp)
>
> copy from (file=testcd)
>
> to (file=uid.filename
>
> disp=rpl)
>
> pend

# Titan System HST

## Titan System Hot Site Test Environment

The Data Center's DR program is designed to support only a subset of the
full production version of the z/OS operating environments for Titan.
The objective of the DR program is to support only the essential
operations and functions of application participating in the disaster
recovery program. The following is the hot site system environment for
Titan:

*DASD* -- All essential system volumes, data base volumes, private
volumes designated by an application, special disaster recovery volumes,
DISR01, DIST00, DIST01, and DIST02, and **[all]{.underline}** public
volumes will be restored.

*Tapes* - None of the tapes from the Titan system production tape
library will be taken to the hot site for HSTs. Copies of the Titan
system backup tapes are sent to restore the Titan system at the hot
site. Any tapes that are required to run the applications should remain
at the Data Center. Only copies should be used for the hot site tests.
Some scratch tapes will be available at the hot site and may be
[temporarily]{.underline} assigned to users during the HST exercises.

*NJE* - NJE services are not part of the Center's DR system, but
depending on customer needs, these services may be available during
future HSTs.

*Printing from the Hot Site* - The most reliable method for obtaining
printed output from a disaster test is to print it on a network attached
printer in your office using the VPS facility. Any output not printed
before the end of the test will be purged.

*File Transfers* -- Files may be transferred to/from the DR system using
the file transfer protocol (FTP). To perform an FTP, logon to
drftp.titan.nih.gov.

*Post-Test Examination of Test Results* - FTP and VPS printing provide
you two methods for saving test data to examine at your leisure
following a test.

*Connectivity* -- The only connectivity will be via TCP/IP. Only the
TCP/IP connections to Connect:Direct will be available.

## Ensuring Critical Files are Available for the HST

Prepare backup and restore procedures for files that are not included in
standard Titan system backup procedures. If you have tapes to be sent to
the hot site for your test, you will be notified when to send them to
the Data Center for shipment.

## Ensuring User Profiles Will be Restored at the Hot Site

Since all public volumes are restored, all user profiles should also
have been restored for the test. If for some reason this was not the
case for previous tests, prior to the hot site test
Recovery-Point-in-Time, log onto the Titan production system, and move
the affected profiles and data sets to a volume that is restored at the
hot site, either a non-public volume that is restored for the
application, or any of the disaster recovery volumes -- DISR01, DIST00,
DIST01, DIST02. The ISPF profile uses either of the following naming
conventions: \$iii.ISPF.ISPPROF or aaaaiii.ISPF.ISPPROF, and the WYLBUR
profile uses the following naming convention: aaaaiii.@WYLBUR.PROFILE,
where "\$iii" is the userid and "aaaaiii" is the account/initials.

Examine each data set referenced in member TSLOGON (if this member
exists in your ISPF profile) and in your WYLBUR profile to ensure they
are not stored on a public volume. Any data set on a volume beginning
with "PUB", "DS", or "MIGRAT", is located on a public volume.

Once moved, you should not need to move the profiles and data sets
again.

A Rexx exec, DRMOVE, can be used to move the ISPF profile dataset to
DIST01. Before executing the exec, exit ISPF. The exec can be used to
move your profiles or other user's profiles for which you have the
proper RACF authorities. From the ready prompt enter DRMOVE. You will be
prompted to either hit the Enter key to move your profile dataset or
enter a userid for the profile that is to be moved. The exec only allows
you to move the profile for one user at a time.

NOTE: This command only moves the profile dataset; referenced datasets
will have to be moved using standard commands.

There are numerous methods for moving data sets, but the following
process should work for both sequential and partitioned (PDS) data sets.
Assuming the ISPF profile, \$iii.ISPF.ISPPROF, is to be moved to DIST01,
from the TSO READY prompt, enter the commands:

> COPY ISPF.ISPPROF ISPF.ISPPROF.NEW VOLUME(DIST01) UNIT(3390)
>
> RENAME ISPF.ISPPROF ISPF.ISPPROF.OLD
>
> RENAME '\$iii.ISPF.ISPPROF.NEW' '\$iii.ISPF.ISPPROF'

Repeat the above process for all data sets referenced in the profile.

Log on to verify that the new profile works correctly. Delete the old
ISPF profile with the command (from the TSO READY prompt again):

> DELETE ISPF.ISPPROF.OLD

## On the Day of the Hot Site Test

The hot site system will be ready for testing beginning at 8:00 am,
Eastern time on the day scheduled for application testing. It should be
noted, the Data Center production systems will continue to be available
to users during regular business hours on that day. During testing, an
easy way to distinguish the two systems is to think of the NIH
production system as the *home* system and the hot site system as the
*away* system. Any processing occurring on the away system does not
affect the applications and processing running on the home system.

## Logging In

CIT has installed a T1 line from the NIH campus to the vendor hot site.
The NIH terminus of the T1 line is located in another building on the
NIH campus, separate from the Data Center. The T1 line is accessible
from the NIH network, NIHnet.

If on the day of the test, there are problems with this connection, an
Internet connection will be used instead and users will be notified.

To connect to the Titan hot site system:

-   For TN3270, use the hostname: drtn3270.titan.nih.gov

-   For FTP, use the hostname: drftp.titan.nih.gov

At the LOGON screen type the same logon string normally used to access
the NIH Titan production facility.

## Potential Problems That May Be Encountered During the HST

Based on experiences from past HST exercises, the following are common
problems, with recommended solutions, that may possibly occur during hot
site testing.

+--------------------+-------------------------------------------------+
| **Problem**        | **Solution**                                    |
+====================+=================================================+
| ***Error           | Issue the following command at the Ready        |
| encountered when   | prompt:                                         |
| logging on***      |                                                 |
|                    | > DEL ISPF.ISPPROF NSCR                         |
| (because your      |                                                 |
| profile and/or     | Log off and then log on again. The system will  |
| referenced files   | automatically build a new default profile for   |
| are stored on a    | you on the DR system. Note that the new profile |
| volume that has    | contains only standard settings; it will not    |
| not been restored; | contain any non-default settings that you may   |
| you are placed at  | have in your production system profile.         |
| the TSO Ready      |                                                 |
| prompt instead of  |                                                 |
| ISPF)              |                                                 |
+--------------------+-------------------------------------------------+
| ***System catalog  | Issue the following TSO command to delete a     |
| conflict***        | catalog entry for a non-existent data set on    |
|                    | the DR system:                                  |
| (because the       |                                                 |
| referenced data    | DEL data-set-name NSCR                          |
| set is on a volume |                                                 |
| that was not       | Note: Executing this command on the DR system   |
| restored)          | will [not]{.underline} affect data sets on the  |
|                    | NIH production system.                          |
+--------------------+-------------------------------------------------+
| ***Cannot submit   | Using panel 3.4, uncatalog your                 |
| jobs*** (you get   | SYS4.SPFLOG1.LIST data set.                     |
| an error message   |                                                 |
| like "unable to    |                                                 |
| allocate temporary |                                                 |
| data set")         |                                                 |
+--------------------+-------------------------------------------------+
| ***Job is not      | The User Display Facility (UDF) allows you to   |
| processing***      | display information about jobs processed by     |
|                    | ThruPut Manager. UDF is invoked through the     |
|                    | ISPF command TMUSER. You can assign TMUSER to a |
|                    | PF key. To use UDF:                             |
|                    |                                                 |
|                    | 1.  Invoke UDF in IOF using one of two methods: |
|                    |                                                 |
|                    | -   Place the cursor at the job name and press  |
|                    |     the PF key you have assigned to TMUSER.     |
|                    |                                                 |
|                    | > or                                            |
|                    |                                                 |
|                    | -   Enter the TMUSER command on the ISPF        |
|                    |     command line, move the cursor to the job    |
|                    |     name, and press Enter.                      |
|                    |                                                 |
|                    | > You will see one or more Information Summary  |
|                    | > Lines. For example:                           |
|                    | >                                               |
|                    | > GL3005TB(J01143) \_ JB JC JL H                |
|                    | >                                               |
|                    | > The Information Summary Line contains the job |
|                    | > name and job number. It may also contain a    |
|                    | > command line and one or more acronyms         |
|                    | > indicating the ThruPut Manager services       |
|                    | > affecting the job. The most common acronyms   |
|                    | > are:                                          |
|                    |                                                 |
|                    |   -------- -----------------------------------  |
|                    |   DC       Dataset Contention Services          |
|                    |                                                 |
|                    |   JB       Job Binding Services                 |
|                    |                                                 |
|                    |   JC       Job Chaining Services                |
|                    |                                                 |
|                    |   JL       Job Limiting Services                |
|                    |                                                 |
|                    |   JS       Job Setup Services                   |
|                    |                                                 |
|                    |   MH       Multi-Hold Services                  |
|                    |                                                 |
|                    |   RS       Robotic Setup Services               |
|                    |   -------- -----------------------------------  |
|                    |                                                 |
|                    | > Highlighted acronyms indicate that the        |
|                    | > service is causing the job to be held.        |
|                    | >                                               |
|                    | > The line may also contain the letters **H**   |
|                    | > or **D**, showing whether the job has been    |
|                    | > held or deferred by ThruPut Manager.          |
|                    |                                                 |
|                    | 2.  To see more information about an acronym,   |
|                    |     place the cursor under the acronym and      |
|                    |     press Enter.                                |
|                    |                                                 |
|                    | 3.  You can enter the following commands on the |
|                    |     command line:                               |
|                    |                                                 |
|                    | -   **D** to see job details                    |
|                    |                                                 |
|                    | -   **V** to see which volumes are used         |
+--------------------+-------------------------------------------------+
| ***Job fails***    | The process discussed below may be used at the  |
| (because the       | hot site to restore your data set from one of   |
| referenced data    | the FDR/ABR full volume dump tapes sent to the  |
| set was not        | hot site.                                       |
| restored)          |                                                 |
|                    | **[Preparation]{.underline}**                   |
|                    |                                                 |
|                    | You will need the following information:        |
|                    |                                                 |
|                    | -   The full name of the data set to be         |
|                    |     restored                                    |
|                    |                                                 |
|                    | -   The volume the data set was on (available   |
|                    |     from the ISPF screen 3.4)                   |
|                    |                                                 |
|                    | **[Using the FDR Primary Options                |
|                    | Menu]{.underline}**                             |
|                    |                                                 |
|                    | 1.  On the ISPF Primary Option Menu, select     |
|                    |     option C.                                   |
|                    |                                                 |
|                    | 2.  On the Additional Products menu, select     |
|                    |     option A.\                                  |
|                    |     \                                           |
|                    |     The FDR Primary Options Menu is displayed.  |
|                    |                                                 |
|                    | **[Creating Job Statements]{.underline}**       |
|                    |                                                 |
|                    | If this is the first time you have used FDR to  |
|                    | restore a data set, you must create job         |
|                    | statements for the print request and the        |
|                    | restore request. If you have previously         |
|                    | restored a data set, you can skip this section. |
|                    |                                                 |
|                    | 1.  On the FDR Primary Options Menu, select     |
|                    |     option J.                                   |
|                    |                                                 |
|                    | 2.  Modify the two sample job statements for    |
|                    |     your jobs.                                  |
|                    |                                                 |
|                    | 3.  Press PF3 to return to the FDR Primary      |
|                    |     Options Menu.                               |
|                    |                                                 |
|                    | ######### **[Determin                           |
|                    | ing the Backup Generation Number]{.underline}** |
|                    |                                                 |
|                    | In order to do the restore you will need to     |
|                    | know the generation number of the backup.       |
|                    |                                                 |
|                    | There are two methods to do this.               |
|                    |                                                 |
|                    | ***The first method:***                         |
|                    |                                                 |
|                    | 1.  On the FDR PRIMARY OPTIONS MENU, select     |
|                    |     option 1 (reports).                         |
|                    |                                                 |
|                    | 2.  On the FDRABR REPORT PANEL, enter option 2, |
|                    |     the dataset name, and the original disk     |
|                    |     volume that it was on, and then hit enter.  |
|                    |                                                 |
|                    | 3.  This should produce one or more screens of  |
|                    |     output. Look for lines of the following     |
|                    |     format:                                     |
|                    |                                                 |
|                    | BKD(00)-2004.206 SFX-C1004200 FN-0040           |
|                    | VOLS-E00427                                     |
|                    |                                                 |
|                    | > Each such line of output represents one       |
|                    | > backup of the disk containing your dataset.   |
|                    | > Only one set of backups will be available at  |
|                    | > the disaster site, so you need to identify    |
|                    | > which backup is available. To do this,        |
|                    | > examine the day of year of the backup (206 in |
|                    | > the example). You need to find the line of    |
|                    | > output which has a day of year equal to a     |
|                    | > date falling on the weekend that the point in |
|                    | > time backups were taken.                      |
|                    |                                                 |
|                    | Now examine the string of characters following  |
|                    | 'SFX-' (C1004200 in the example line). The      |
|                    | 3^rd^ through the 6^th^ characters in the       |
|                    | string (0042 in the example line) is the backup |
|                    | generation number. You will need to know it in  |
|                    | order to restore your dataset.                  |
|                    |                                                 |
|                    | ***The second method:***                        |
|                    |                                                 |
|                    | If you have trouble determining the backup day  |
|                    | of year or the backup generation number, call   |
|                    | the DR User Support Team. You will be contacted |
|                    | by a member of the DR restore team, who should  |
|                    | be able to tell you the backup generation       |
|                    | number for your disk from their listings of the |
|                    | backups.                                        |
|                    |                                                 |
|                    | **[Restoring a Data Set]{.underline}**          |
|                    |                                                 |
|                    | 1.  On the FDR Primary Options Menu, select     |
|                    |     option 2.                                   |
|                    |                                                 |
|                    | 2.  On the FDRABR Restore Panel enter the       |
|                    |     following information:                      |
|                    |                                                 |
|                    | +---------------------+---------------------+   |
|                    | | ### Option          | **Data To Enter**   |   |
|                    | +---------------------+---------------------+   |
|                    | | SELECT OPTION       | B                   |   |
|                    | +---------------------+---------------------+   |
|                    | | COPY OPTION         | 2                   |   |
|                    | +---------------------+---------------------+   |
|                    | | PROCESS OPTION      | B                   |   |
|                    | +---------------------+---------------------+   |
|                    | | GENERATION          | the generation      |   |
|                    | |                     | number (obtained    |   |
|                    | |                     | using the above     |   |
|                    | |                     | procedure)          |   |
|                    | +---------------------+---------------------+   |
|                    | | CYCLE               | 0                   |   |
|                    | +---------------------+---------------------+   |
|                    | | ORIGINAL VOLUME     | volume serial       |   |
|                    | | SERIAL              | number for the      |   |
|                    | |                     | original data set   |   |
|                    | +---------------------+---------------------+   |
|                    | | NEW VOLUME SERIAL   | DSP101              |   |
|                    | +---------------------+---------------------+   |
|                    | | OTHER DSNAME/FILTER | original data set   |   |
|                    | |                     | name                |   |
|                    | +---------------------+---------------------+   |
|                    | | OTHER NEW NAME      | new data set name   |   |
|                    | +---------------------+---------------------+   |
|                    |                                                 |
|                    | > You may need to scroll down to see the OTHER  |
|                    | > NEW NAME field.                               |
|                    |                                                 |
|                    | 3.  Press Enter.                                |
|                    |                                                 |
|                    | > The data set will be restored to the new data |
|                    | > set name.                                     |
|                    |                                                 |
|                    | **[Cataloging the Restored Data Set Under the   |
|                    | Original Name]{.underline}**                    |
|                    |                                                 |
|                    | After the data set is restored, you will        |
|                    | probably want to change the new data set name   |
|                    | to the original data set name. You must first   |
|                    | uncatalog the original data set because it is   |
|                    | still listed in the catalog even though the     |
|                    | data set does not exist.                        |
|                    |                                                 |
|                    | 1.  Use the ISPF screen 3.4 to list the data    |
|                    |     set.                                        |
|                    |                                                 |
|                    | 2.  Enter the UNCATALOG line command in the     |
|                    |     command area next to the original data set  |
|                    |     name.                                       |
|                    |                                                 |
|                    | 3.  Enter the RENAME line command in the        |
|                    |     command area next to the new data set name. |
|                    |                                                 |
|                    | 4.  Rename the data set to the original name.   |
+--------------------+-------------------------------------------------+
| ***Output from DR  | Have your firewall group permit LPR traffic     |
| Titan won't print  | from 165.112.213.229 to enter your network.     |
| via VPS to your    |                                                 |
| network printer*** |                                                 |
+--------------------+-------------------------------------------------+
| ***Have accessed   | To connect to the Titan hot site system:        |
| the Titan home     |                                                 |
| system when        | -   For TN3270, use the hostname:               |
| intended to access |     drtn3270.titan.nih.gov                      |
| the Titan hot site |                                                 |
| system***          | -   For FTP, use the hostname:                  |
|                    |     drftp.titan.nih.gov                         |
+--------------------+-------------------------------------------------+
| ***Denial of       | Treat the tape as a foreign tape at the hot     |
| access to a        | site by adding                                  |
| tape***            |                                                 |
|                    | EXPDT = 98000 to the DD card.                   |
| (because the tape  |                                                 |
| was created after  |                                                 |
| the                |                                                 |
| Recov              |                                                 |
| ery-Point-in-Time; |                                                 |
| the assignment to  |                                                 |
| the                |                                                 |
| user/application   |                                                 |
| is not recorded in |                                                 |
| the tape library   |                                                 |
| system at the hot  |                                                 |
| site.)             |                                                 |
+--------------------+-------------------------------------------------+

# EOS Hot Site Test

## EOS Hot Site Test Environment

The Data Center's DR program is designed to support only a subset of the
full production version of the EOS operating environments. The objective
of the DR program is to support only the essential operations and
functions of applications participating in the disaster recovery
program.

DCSS has contracted for multiple ALPHA and Sun servers to host the
participating applications. These servers will be configured with the
current versions of the Alpha Tru64 UNIX and Solaris operating systems,
and Oracle database.

The hot site server configurations have sufficient memory and disk space
to accommodate the execution of the applications. All application and
database files that have been identified by the application owner as
being required, will be restored to the hot site servers.

## Ensuring Critical Files are Available for the HST

Prior to the hot site test, DCSS will request the list of data that is
to be backed up for the test.

**NOTE:** **To ensure an efficient and error-free restoration of your
files, include only those files necessary for application processing,
e.g., application files, database files, configuration files. Do not
include old log files, report files, or other files that can easily be
recreated.**

## Accounts on the Hot Site ALPHA Server

Prior to the HST, DCSS will request the names and IP addresses of all
users who will be logging on [directly]{.underline} to the hot site
ALPHA Server. DCSS will set up accounts on the hot site host for the
test and will notify the testers of the login IDs and passwords.

## Considerations for Oracle 

In order to minimize the possibility of inadvertently connecting to the
wrong databases while performing DR testing, the following are
suggestions to consider:

For client (remote) machines connecting to the DR Oracle databases:

-   Ensure that the sqlnet.ora file is removed or the references to
    production Oracle Names Servers are removed.

-   Ensure that tnsnames.ora entries that will be used for testing are
    modified. Either add a new database alias (e.g., that starts with
    DR), or modify the existing database alias definition to point to
    the DR host. See the section heading, Connecting to a Disaster
    Recovery Database Instance, below for further instructions on this.

For host (local) DR database machines:

-   Determine whether you want to have any database alias definitions
    removed or changed.

-   If using database links or specifying aliases during connection that
    use aliases, request that the database tnsnames.ora file be changed
    to point to the appropriate databases (e.g., host, sid, port).

-   If testing using database links, either request a change to the
    database alias in the host tnsnames.ora file,or, if the database
    links were created with host name parameter or password in the using
    clause, recreate the link.

-   If appropriate, alter database users to use a non-production
    password.

Client changes for DR Oracle Application Server:

-   Modify any hard coded URL\'s pointing to the production database
    within the application to point to the DR site.

-   For 9iAS Release 2, customers will need to configure DAD\'s to point
    to the DR database, after CIT Oracle System staff install and
    configure the middle tier software to point to the DR database.

-   For 9iAS Release 1, if customers want the database alias changed on
    the middle tier machine, the DAD\'s will need to be modified to
    point to the DR database alias, after CIT System staff has restored
    and configured the middle tier software to point to the DR database.

The following changes will be made by CIT Oracle System staff on the DR
machines.

For the DR Oracle Database instances:

-   Modify the host name in listener.ora file to point to DR host.

-   Remove the sqlnet.ora entries pointing to the production Oracle
    Names Servers.

-   Verify with customers regarding preferred changes they want to the
    DR database local tnsnames.ora file.

-   Verify customers want the database alias changed in addition to the
    host value.

-   Verify with customers regarding the removal or modification of
    additional database aliases in the tnsnames.ora file.

For the DR Oracle Middle Tier instances:

-   Modify the tnsnames.ora file to specify the DR host for the database
    alias in \$IASHOME and \$6iSERVER_HOME.

-   Modify the httpd.conf file to specify the DR virtual host.

-   If URL Redirection is used, make sure it is pointing to DR site.

-   Modify the httpd.conf file to have the LISTEN parameter pointing to
    the virtual IP address of the DR site.

-   Modify the oem.conf file to have the virtual host parameter pointing
    to DR site.

-   If Report Server is used, set the DISPLAY parameter point to IP
    address of the DR host.

## On the Day of the Hot Site Test

The hot site system will be ready for testing beginning at 8:00 am,
Eastern time on the day scheduled for application testing. It should be
noted, the Data Center production systems will continue to be available
to users during regular business hours on that day. During testing, an
easy way to distinguish the two systems is to think of the NIH
production system as the *home* system and the hot site system as the
*away* system. Any processing occurring on the away system does not
affect the applications and processing running on the home system.

## Hot Site Hostname

CIT has installed a T1 line from the NIH campus to the vendor hot site.
The NIH terminus of the T1 line is located in another building on the
NIH campus, separate from the Data Center. The T1 line is accessible
from the NIH network, NIHnet.

If on the day of the test, there are problems with this connection, an
Internet connection will be used instead and users will be notified.

You will use a special hostname for the DR test. The DR hostname has
been constructed by putting "dr" at the beginning of your production
hostname. For example: **dreos.cit.nih.gov** is the DR hostname for
eos.cit.nih.gov.

## Logging In

Direct logins use *ssh*.

## Connecting to a Disaster Recovery Database Instance

If you will be using SSH to connect to your Disaster Recovery instance,
you do not have to take any special steps. The tnsnames.ora file on the
dreos.nih.gov machine will be updated for your instance.

If you will be using a client/server connection, you can do either of
the following:

1)  Change the host parameter in the tnsnames.ora file on your client
    machine for the alias you will be using for the disaster recovery
    test. Change the statement:

> HOST = \<*hostname*\>
>
> to
>
> HOST = dreos.cit.nih.gov
>
> The entry should resemble the example in 2) below:

2)  Create a new alias "drtest" in the tnsnames.ora file to be used for
    disaster recovery testing:

> drtest =
>
> (DESCRIPTION =
>
> (ADDRESS = (PROTOCOL = TCP)(HOST = dreos.cit.nih.gov)(PORT = \<port
> number\>))
>
> (CONNECT_DATA = (SID = test))
>
> )