# Presentation: 800207

## Htar

**H**pss **T**ape **Ar**chiver
- Client API-based interface written by Mike Gleicher
-             Originally commissioned for LLNL in 2000
-             Now available as part of the HPSS Offering
- A File Bundling and manipulation tool that conforms to
-             standard tar requirements:
-             Bundles small files on a platform into large files in
-             storage.  The Htar is created in the storage system,
-             each file is transferred directly into the Htar in storage.

## Why use Htar 

- Powerful command set that makes it easy to recursively archive
-          files to storage:  whole directories, files in lists, etc.with
-          one command.
- A users directory structure is maintained in the “htar” file.
- Multiple files transfer faster with htar:
-         HTAR –cvf htar.tar ./mydir is optimized for simultaneous
-                       transfer
-         HSI put –r ./mydir  is recursive..... But transfers files one
-                       at a time .

## Htar@NERSC

- Authentication:
- Htar uses the same .hsipw file as HSI for authentication to NERSC
-           storage systems.  If you currently have a .hsipw file Htar
-           will use it.  If you don’t you will be prompted for your
-           dce combo, and if it’s successful a .hsipw will be created
-           in ~HOME.
- Availability:
-  Installed on Seaborg /usr/common/mss/bin and
-                 on the PDSF /usr/local/bin

##  Htar Definitions 

**Archive File** - The large “tar” file that Htar creates out of files
**Member File** - one of the files contained in the Htar Archive File.
**Index File** - a catalog of the Member Files contained in the
-                     Archive File
**Consistency File** - the file at the end of an Archive File used to
-                      verify the consistency of the Archive File and the
-                       Index File.

## Instorage

**Htar -cvf myhtar.tar**
- creates the following in storage**:**
-              the **archive file** called  myhtar.tar
-              the **index file** called     myhtar.tar.idx
-              a **consistency file** inside the archive file
-                 (looks like a member file but is treated differently)
-              called      /tmp/HTAR_CF_CHK.....
- This consistency file can be used to verify the consistency of
- the index file and the archive file.  It will not be extracted unless
- explicitly specified using the Htar command.
- (it would be extracted using the tar command)

## Htar behavior

**Absolute path names:**
**       ** pathnames beginning with “/” can be used when creating
-         the htar...when files are extracted from the htar the
-         leading “/” is removed.  All extracted files use pathnames
-         that are relative to the current working directory.
- Htar makes use of the **TMPDIR** environment variable when creating
-          temporary files.  If TMPDIR is not set in the environment
-          then “/tmp” is used.
- There is **no default device**  for Htar the -f flag must be used.

## Htar and tar

**Tar Similarities**:
- Htar conforms to standard tar requirements.  An Htar Archive
-     File can be manipulated using the tar utility.  File and
-     directory name lengths and file sizes must conform to
-      tar restrictions (POSIX 1003.1 USTAR).  Buffer path to
-     155 chars, name buffer to 100 bytes.  Member file size <= 8GBs.
**Tar Differences**:
- Htar  Can’t append a new Member File to an existing
-      Archive File.  Htar Can’t update/replace Member Files
-      in an existing Archive File.
- Htar creates an index  .idx file in the same dir as the archive file.
-      and a consistency check file inside the Htar archive file

## Htar examples

**Htar -cvf files.tar file1 file2**
-       writes file1 and file2 into a new archive called files.tar in the
-       current hpss home directory.
**Htar -cvf /nersc/user/homedir/htar.tar ./dir1**
-        writes all the files in the current dir1 directory into a new
-        archive file called htar.tar in the hpss directory
-        /nersc/user/homedir
**Htar -tvf htar.tar**
**       **list files in the archive file htar.tar in the current hpss
-        homedir.

## Htar more examples

**Htar -xvf files.tar**
-       extract all the files from the archive file files.tar into the
-       current directory.
**Htar -xf proj1.tar project1/src**
-        extract all files in project1/src directory of the archive file
-        proj1.tar into the current directory
**Htar -xf file1 proj1.tar**
-        extract the individual file :file1: from the archive file proj1.tar

## Htar extensions

**Htar -X -f /nersc/user/hpsshomedir/tarfile**
-         creates an index for a tarfile in hpss.  This file can
-         now be accessed using htar.
**Htar -X -E -f localtarfile**   : creates an htar index for a
-          local tar file.
**Htar -K -f htartest.htar  :  **use the consistency file
-         to verify the index and the archive file.

## Recommended Usage 

- Htar should replace all current tar transfers piped to storage.
- Htar should be used with data that is not changing.  Htar makes two
-         passes thru the data first to generate the index file, second pass
-         to transfer the data.  If files change before they are transferred
-         and after the index is created a warning is issued and the file is
-         truncated or padded to match the index.
- Htar should be used to bundle small files
- Htar should be used to archive files that are not frequently
-         referenced  (purely archival)

##  Usage for Performance 

- Htar should be used with **collections of data** that will reaccessed
-          as a collection.  These Htar files should be < 6GBs.
- Read access to files in storage:
- 1 second  = 86400 accesses/day at XXMBs/sec
- 30 seconds = 2880 accesses/day at XXMBs/sec
- 60 seconds = 1440   .../etc
-         The Htar will travel in the archive as one file to and from
-          disk and tape.  Access to any file in the Htar will stream the
-          Htar to disk improving all subsequent accesses to the data
-          to 1 second or less.