# Presentation: 314280

## ARCHIVE PROCESSING

**ECS Release 5A Training**

- 625-CD-005-001

## Overview of Lesson

**Introduction**
**Archive Processing Topics**
**Archive Hardware and Software**
**Start and Shut Down AMASS**
**Archive Resources and Management**
**Insert and Retrieve Data**
**Load Archive Media**
**Backup and Restore Archive Data**
**AMASS Graphical User Interface**
**Monitor Archive System and**
**   ****Respond to Fault Notification**
**Practical Exercise**

## Objectives

**Overall:  Proficiency in Archive Processing**
**Describe Archive Manager roles and responsibilities**
**Identify and describe Archive storage resources**
**Start and shut down AMASS**
**Describe archive storage element relationships and archive resource management**
**Purge expired files from pull monitor cache**
**Load and unload cartridges**
**Insert/retrieve data**
**Delete files from the archive**
**Backup archive data**
**Restore archive data**
**Use the AMASS Graphical User Interface**
**Monitor archive system and perform fault notification**

- 625-CD-005-001

## DAAC Archive Manager Interfaces

**DAAC Archive Manager Interfaces**

- 625-CD-005-001

## EMASS Archival Management and Storage System 

- 625-CD-005-001

## STK PowderhornLibrary Storage Module (LSM)

- 625-CD-005-001

## EMASS Archive Tape Cartridge

-  3 MB/sec uncompressed sustained data transfer rate
-  6.6 MB/sec compressed sustained data transfer rate
-  2.4 GB capacity per cartridge (compressed)
-  36-track read/write capabilities
-  High reliability (45,000 hr MTBF*)
- *MTBF = Mean Time Between Failures

- 625-CD-005-001

## EMASS Archive Optical Cartridge

- 625-CD-005-001

## STK D-3 Tape Cartridge

- 625-CD-005-001

## AMASS Control Path

- 625-CD-005-001

## Automated Cartridge System Library Software (ACSLS)

- 625-CD-005-001

## Automated Cartridge System Library Software (ACSLS) (Cont.)

**General Command Syntax:**
**command  type_identifier  state  [options]**
**Frequently used commands:**
**query, vary, enter, eject**
**Utilities:**
**bdb.acsss -- back up the ACSLS database**
**kill.acsss -- terminate ACSLS**
**rc.acsss -- start and recover ACSLS**
**rdb.acsss****	**** -- restore the ACSLS database**
**volrpt -- create a volume report**
**db_command -- start/stop Oracle database**
**User Ids:**
**acssa -- enter commands**
**acsss -- run utilities from UNIX**

- 625-CD-005-001

## Distributive AML Server (DAS) Software Monitoring Commands

- 625-CD-005-001

## Sources and Uses of Archive Data

- 625-CD-005-001

## EMASS AML Control Panel

- 625-CD-005-001

## STK Control Panels

- 625-CD-005-001

## Storage Element Relationships

**Server Key (e.g., DRP1_OPS), mapped to a Cell Directory Service (CDS) entry (e.g., EcDsStArchiveServer DRP1_OPS)**

**CDS group may include multiple Logical Volume Groups (LVGs), each with a specified path, mapped to a volume group in the physical archive**

**EcDsStArchiveServer DRP1_OPS**

**   ****VG1 at /*****path 1***
***   *****VG2 at /*****path 2***
**   ****VG3 at /*****path 3***
**         ****.**
**         ****.**
**         ****.**

- 625-CD-005-001

## Storage Management:Storage Config. Tab

**Storage Management:****Storage Config. Tab**

- 625-CD-005-001

## Archive-Related Applications

**Servers (must be running)**
**Science Data Server (SDSRV)**
**Storage Management (STMGT) Servers**
**Staging Disk Server**
**Staging Monitor Server**
**Archive Server**
**Ingest FTP Server**
**Distribution FTP Server**
**D3 Tape Server**
**8mm Tape Stacker Server**
**Data Distribution (DDIST) Server**
**DSS Graphical User Interface (GUI)**

- 625-CD-005-001

## DSS Science Data Server GUI:Data Types Tab, with Add Dialog

- 625-CD-005-001

## Archive Resource Management

**For Science Software Integration and Test (SSI&T)**
**At the time an Earth Science Data Type (ESDT) is loaded, its Archive ID is identified**
**e.g., DRP1_OPS:VG1**
**ESDTs may be uniquely assigned to logical volume groups**
***Note*****:  At a later time (Drop 5A?) it is planned that the logical volume group will be specified as the ESDT short name with the version ID as an extension (e.g., MODIS01.001)**

- 625-CD-005-001

## Storage Management:Resource Schedule Tab

**Storage Management:****Resource Schedule Tab**

- 625-CD-005-001

## Storage Management:Cache Stats. Tab

- 625-CD-005-001

## Insert Data Into the Archive

- 625-CD-005-001

## System ManagementRequests Window

- 625-CD-005-001

## System ManagementFilter Requests Window

- 625-CD-005-001

## Data Server:Distribution Requests Window

- 625-CD-005-001

## Distribution Filter Requests Window

**Distribution Filter Requests Window**

- 625-CD-005-001

## Retrieval of Data from the Archive

**Largely automated process in response to data distribution requests**
**data orders from scientists or other ECS end users**
**one-time orders**
**standing orders placed as subscriptions for acquiring data**
**data requests from other ECS sites**
**cross-DAAC orders for end users.**
**data needed as input for processing at other sites (subscriptions placed for ingest by those sites)**
**internal requests for data needed for processing**
**DSS Distrib’n Requests window to monitor **

- 625-CD-005-001

## Loading/Removing Archive Media

**EMASS AML:  Entry Interface Facility (EIF)**
**30 Tapes**
**22 Optical cartridges**
**STK Powderhorn:  Cartridge Access Port (CAP)**
**21 Tapes**
**Automatic loading (recommended for most loading)**
**Manual loading (available for initial loading or other bulk loading)**

- 625-CD-005-001

## Archive Backups

**Programmatic backups are part of design requirements**
**Active archive copy (use *****Archive ID*****)**
**Local backup storage copy (use *****Backup ID*****)**
**Off-site backup storage copy (use *****Offsite ID*****)**
**Archive manager needs to create volume group for offsite backups**
**Selection of data for backup dependent on factors such as feasibility of recovery by other means (e.g., re-ingest, reprocess)**
**Each site is responsible for arranging its own secure offsite storage**

- 625-CD-005-001

## Some AMASS Commands

***vollist***
**Lists all volumes and current status**
***dirfilelist***
**lists files under a specified directory**
***volfilelist***
**lists all files on a specified volume**
**Others:  See *****AMASS System Administrator’s Guide***

- 625-CD-005-001

## Creating a Backup for AMASS

**AMASS tape format is proprietary, designed for speed of access**
** *****vgexport -q***** command creates an ASCII file that can be used with the tapes and *****vgimport***** command to recover the stored data**

- 625-CD-005-001

## Replace Backup Volume (Volume 1)

**AMASS database backup is stored in the archive on Volume 1**
**AMASS issues warning when Backup Volume is nearly full (95%)**
**When warning message is received, install new Backup Volume and perform a full backup**
**If Backup Volume gets full during attempted backup, backup will fail, necessitating replacement and full backup**

- 625-CD-005-001

## Manual Backup of Archive Data

**Necessary when a backup data tape is used and must be replaced**
**Data restoration using a backup data tape is achieved by inserting the tape into the archive**
**To replace the tape requires manual creation of a copy**

- 625-CD-005-001

## Restoring Archive Data

**Copy from backup to primary (using UNIX)**
**Reprocess lower-level data to obtain lost higher-level product**
**Request replacement from data provider**
**Restoring AMASS/ACSLS database**

- 625-CD-005-001

## AMASS GUI

- 625-CD-005-001

## AMASS GUI: Modify A VG Screen

- 625-CD-005-001

## AMASS GUI: Modify A Volume Screen

- 625-CD-005-001

## Storage Management:Storage Events Tab

**Storage Management:****Storage Events Tab**

- 625-CD-005-001

## AMASS Monitoring Commands, Utilities, and Scripts

** *****sysperf***** is a command to display AMASS I/O activity**
** *****amass_log***** is a script permitting display of AMASS messages from the system log**
** *****quedisplay***** is a library utility used to view the AMASS queue and diagnose problems**
** *****medialist**** ***shows robot’s view of media and their slot locations**
** *****mediamove***** can establish synchrony between *****quedisplay***** and *****medialist***
**There are other utilities and scripts described in *****AMASS System Administrator’s Guide***

- 625-CD-005-001

## Sample Output from quedisplay

- 625-CD-005-001

## Recovery from Failure to Store Data

**Storage failure may result from file copy errors**
**Network problems**
**Mount point problems**
**AMASS unavailable (e.g., off line)**
**Use *****amass_log***** script to display errors**
**Corrective action from *****AMASS System Administrator’s Guide***
**Resume action**

- 625-CD-005-001

## Checksum De-activation

**System design incorporates calculation of checksum on granule insertion**
**Supports detection of data corruption**
**Comparison of original checksum with checksum calculated on data retrieval from the archive**
**De-activation of checksum calculation may improve system throughput**
**Compromises detection of data corruption**
**Partial alleviation by comparison of checksums calculated on sequential data retrievals**

- 625-CD-005-001