# Presentation: 674211

## Optimizing shared access 

**to tertiary storage**** **
**Arie Shoshani**
** ****Alex Sim**
**July 10, 2001**
**Scientific Data Management Group**
**Computing Science Directorate**
**Lawrence Berkeley National Laboratory**

## Outline

**Short High Energy Physics overview ****(of data handling problem)**
**Description of the Storage Coordination System**
**File tracking**
**File sharing between file access requests**
**coordination of “file bundles”**
**The Hierarchical Storage Manager (HRM)**
**tertiary storage queuing and tape coordination**
**transfer time for query estimation**

## Optimizing Storage Management for High Energy Physics Applications

**Data Volumes for planned ****HENP experiments**

**STAR****: Solenoidal Tracker At ****RHIC**
**RHIC****: Relativistic Heavy Ion Collider **

## Particle Detection Systems

- STAR detector at RHIC

- Phenix at RHIC

## Result of Particle Collision (event)

## Typical Scientific Exploration Process

**Generate large amounts of raw data**
**large simulations**
**collect from experiments**
**Post-processing of data**
**analyze data (find particles produced, tracks)**
**generate summary data**
**e.g. momentum, no. of pions, transverse energy**
**Number of properties is large (50-100)**
**Analyze data**
**use summary data as guide**
**extract subsets from the large dataset**
**Need to access events based on partial****properties specification (range queries)**
**e.g. ((0.1 < AVpT < 0.2) ^ (10 < Np < 20)) v (N > 6000)**
**apply analysis code**

##  STAR experiment
108 events over 3 years
1-10 MB per event: reconstructed data
events organized into 0.1 - 1 GB files
1015 total size
106 files, ~30,000 tapes (30 GB tapes)
Access patterns
Subsets of events are selected by region in high-dimensional property space for analysis
10,000 - 50,000 out of total of 108 
Data is randomly scattered all over the tapes
Goal: Optimize access from tape systems 

**10****8**** events over 3 years**
**1-10 MB per event: reconstructed data**
**events organized into 0.1 - 1 GB files**
**10****15**** total size**
**10****6**** files, ~30,000 tapes (30 GB tapes)**
**Access patterns**
**Subsets of events are selected by region in ****high-dimensional property space for analysis**
**10,000 - 50,000 out of total of 10****8**** **
**Data is randomly scattered all over the tapes**
**Goal: Optimize access from tape systems **

**Size of Data and Access Patterns **

## EXAMPLE OF EVENT PROPERTY VALUES

**I  event 1**
**I  N(1) 9965**
**I  N(2) 1192**
**I  N(3) 1704**
**I  Npip(1) 2443**
**I  Npip(2) 551**
**I  Npip(3) 426**
**I  Npim(1) 2480**
**I  Npim(2) 541**
**I  Npim(3) 382**
**I  Nkp(1) 229**
**I  Nkp(2) 30**
**I  Nkp(3) 50**
**I  Nkm(1) 209**
**I  Nkm(2) 23**
**I  Nkm(3) 32**
**I  Np(1) 255**
**I  Np(2) 34**

**I  Np(3) 24**
**I  Npbar(1) 94**
**I  Npbar(2) 12**
**I  Npbar(3) 24**
**I  NSEC(1) 15607**
**I  NSEC(2) 1342**
**I  NSECpip(1) 638**
**I  NSECpip(2) 191**
**I  NSECpim(1) 728**
**I  NSECpim(2) 206**
**I  NSECkp(1) 3**
**I  NSECkp(2) 0**
**I  NSECkm(1) 0**
**I  NSECkm(2) 0**
**I  NSECp(1) 524**
**I  NSECp(2) 244**
**I  NSECpbar(1) 41**
**I  NSECpbar(2) 8**

**R  AVpT(1) 0.325951**
**R  AVpT(2) 0.402098**
**R  AVpTpip(1) 0.300771**
**R  AVpTpip(2) 0.379093**
**R  AVpTpim(1) 0.298997**
**R  AVpTpim(2) 0.375859**
**R  AVpTkp(1) 0.421875**
**R  AVpTkp(2) 0.564385**
**R  AVpTkm(1) 0.435554**
**R  AVpTkm(2) 0.663398**
**R  AVpTp(1) 0.651253**
**R  AVpTp(2) 0.777526**
**R  AVpTpbar(1) 0.399824**
**R  AVpTpbar(2) 0.690237**
**I  NHIGHpT(1) 205**
**I  NHIGHpT(2) 7**
**I  NHIGHpT(3) 1**
**I  NHIGHpT(4) 0**
**I  NHIGHpT(5) 0**

**54 Properties,  as many as 10****8**** events**

## Opportunities for optimization

**Read only events qualified for a query from a file (avoid reading irrelevant events)****=> exact index over all properties**** **
**Share files brought into cache by multiple queries****=> look ahead for files needed and cache management**** **
**Read files from same tape when possible****=> coordinating file access from tape**

**Opportunities for optimization**

## The Storage Access Coordination System (STACS)

**The Storage Access Coordination System (STACS)**

- Query
- Estimator
- (QE)

- Hirarchical
- Resource
- Manager
- (HRM)

- Query
- Monitor
- (QM)

-  Query
- estimation /
- execution
- requests

- file caching
- request

- Caching
- Policy
- Module

- File
- Catalog
- (FC)

- Bit-
- Sliced
- index

- file purging

- Disk Cache

- file
- caching

- User’s
- Application

- open,
- read,
- close

## A typical SQL-like Query

**SELECT ***
**FROM star_dataset**
**WHERE 500<total_tracks<1000 & energy<3**

**-- The index will generate a set of files**
**    ****{F****6 ****: E****4****,E****17****,E****44****, F****13 ****: E****6****,E****8****,E****32****, ..., F****1036 ****: E****503****,E****3112****} **
**    ****that the query needs**
**-- The files can be returned to the application**
**    ****in any order**

## File Tracking (1)

## File Tracking (2)

## File Tracking

- query1
- start

- query2
- start

- query3
- start

- All 3
- queries

## Sharing of Files 

**by Multiple Queries**

## File Bundles:Multiple Event Components

## A typical SQL-like Queryfor Multiple Components

**SELECT Vertices, Raw**
**FROM star_dataset**
**WHERE 500<total_tracks<1000 & energy<3**

**-- The index will generate a set of ****bundles**
**    ****{[F****7****, F****16****: E****4****,E****17****,E****44****], [F****13****, F****16****: E****6****,E****8****,E****32****], ...}**
**    ****that the query needs**
**-- The ****bundles**** can be returned to the application**
**    ****in any order**
**-- ****Bundle****: the set of files that need to be in cache**
**   ****at the same time**

## File Weight Policy for ManagingFile Bundles

- File weight (bundle) = 1 if it appears in a bundle,                                = 0 otherwise
- Initial file weight = SUM (all bundles for each query) over all queries
- Example:
- query 1: file FK appears is 5 bundles
- query 2: file FK appears is 3 bundles Then, IFW (FK) = 8

## File Weight Policy for ManagingFile Bundles (cont’d)

- Dynamic file weight: the file weight for a file in a bundle that was processed is decremented by 1
-  Dynamic Bundle Weight

## How file weights are usedfor caching and purging

**Bundle**** caching policy**
**For each query, in turn, cache the bundle****with the most files in cache**
**In case of a tie, select the bundle with the highest weight**
**Ensures that a bundle that include files needed by other bundles/queries have priority**
**File**** purging policy**
**No file purging occurs till space is needed**
**Purge file not in use with smallest weight**
**Ensures that files needed in other bundles stay in cache**

## Other policies

**Pre-fetching policy**
**queries can request pre-fetching of bundles****subject to a limit**
**Currently, limit set to two bundles**
**multiple pre-fetching useful for parallel processing**
**Query service policy**
**queries serviced in Round Robin fashion**
**queries that have all their bundles cached****and are still processing are skipped**

## Managing the queues

**Query**

**Queue**

**Bundle**

**Set**

**File**

**Set**

**Files**

**Being**

**Processed**

**Files**

**in Cache**

## File Tracking of Bundles

**Query 1**
**starts here**

**Query 2**
**starts here**

**Bundle was**
**found in cache**

**Bundle shared**
**by two queries**

**Bundle (3 files)**
**formed, then**
**passed to query**

## Summary

**The key to managing bundle caching and purging policies is ****weight assignment**
**caching - based on ****bundle**** weight**
**purging - based on ****file**** weight**
**Other file weight policies are possible**
**e.g. based on bundle size**
**e.g. based on tape sharing**
**Proving which policy is best - a hard problem**
**can test in real system - expensive, need stand alone**
**simulation - too many parameters in query profile can vary: processing time, inter-arrival time, number of drive, size of cache, etc.**
**model with a system of queues - hard to model policies**
**we are working on last two methods**

## Read files from same tape 

**when possible**

## Queuing File Transfers

**Number of PFTPs to HPSS are limited**
**limit set by a parameter - NoPFTP**
**parameter can be changed dynamically**
**HRM is multi-threaded**
**issues and monitors multiple PFTPs in parallel**
**All requests beyond PFTP limit are queued**
**File Catalog used to provide for each file**
**HPSS path/file_name**
**Disk cache path/file_name**
**File size**
**tape ID**

## File Queue Management

**Goal**
** ****minimize tape mounts**
** ****still respect the order of requests**
** ****do not postpone unpopular tapes forever**
**File clustering parameter - FCP**
**If the file at top of queue is in Tape****i ****and FCP > 1 (e.g. 4) then up to 4 files ****from Tape****i**** will be selected to be ****transferred next**
**then, go back to file at top of queue**
**Parameter can be set dynamically**

**F****1****(T****i****)**

**F****3****(T****i****)**

**F****2****(T****i****)**

**F****4****(T****i****)**

**1**

**2**

**3**

**4**

**5**

**Order**
**of file**
**service**

## File Caching Order fordifferent File Clustering Parameters

**File Clustering Parameter = 1 **

**File Clustering Parameter = 10**

## Transfer Rate (Tr) Estimates

**Need Tr to estimate total time of a query**
**Tr is average over recent file transfers from the****time PFTP request is made to the time transfer completes.  This includes:**
**mount time, seek time, read to HPSS Raid,****transfer to local cache over network**
**For dynamic network speed estimate**
**check total bytes for all file being transferred****over small intervals (e.g. 15 sec)**
**calculate moving average over n intervals****(e.g. 10 intervals)**
**Using this, actual time in HPSS can be estimated**

## Dynamic Display of Various Measurements

## Query Estimate

**Given: transfer rate Tr.**
**Given a query for which:**
**X files are in cache**
**Y files are in the queue**
**Z files are not scheduled yet**
**Let s(file_set) be the total byte size of ****all files in file_set**
**If Z = 0, then**
**QuEst = s(Y’)/Tr**
**If Z = 0, then**
**QuEst = (s(T)+ q.s(Z))/Tr****where q is the number of active queries**

**F****1****(Y)**

**F****3****(Y)**

**F****2****(Y)**

**F****4****(Y)**

**T**

## Reason for q.s(Z)

**20 Queries of length ~20 minutes**
**launched 20 minutes apart**

**20 Queries of length ~20 minutes**
**launched 5 minutes apart**

**Estimate pretty close**

**Estimate bad - request**
**accumulate in queue**

## Error Handling

**5 generic errors**
**file not found**
**return error to caller**
**limit PFTP reached**
**can’t login**
**re-queue request, try later (1-2 min)**
**HPSS error (I/O, device busy)**
**remove part of file from cache, re-queue**
**try n times (e.g. 3), then return error****“transfer_failed”**
**HPSS down**
**re-queue request, try repeatedly till successful**
**respond to File_status request with “HPSS_down”**

## Summary

**Heirarchical Resource Manager**
**insulates applications from transient ****HPSS and network errors**
**limits concurrent PFTPs to HPSS**
**manages queue to minimize tape mounts**
**provides file/query time estimates**
**handles errors in a generic way**
**Same API can be used for any MSS, such****as Unitree, Enstore, Castor, etc.**

## Future Plans

**Package the HRM (CORBA, APIs)**
**Dynamic file catalog Capability**
**Interface HRM to HSI eliminating the need for a file catalog**
**Expand use of HRM to include “write capability”**
**Build upon results of Probe project**
**Large block size, tape striping**
**Provide hints to MPI-IO module**
**Apply to HENP experiments – initially STAR**
**Work with PPDG collaboration pilot (SciDAC)**
**Perform research on “object granularity caching”**