# Presentation: 472138

## Interfacing the 

**Storage Resource Broker (SRB)**
**to the **
**Hierarchical Resource Manager (HRM)**
**Arie Shoshani, Alex Sim (LBNL)**
**Reagan Moore, Bing Zhu (SDSC)**
**PPDG meeting**
**July 13-14, 2000**

## Outline

**HRM role in the Data Grid architecture**
**HRM API description **
**HRM - HPSS system functionality**
**SRB Client calls**
**SRB-HRM architecture**
**Status**

## HRM role in the Data Grid architecture

**The class of Storage Resource Managers (SRMs) includes:**
**HRM: for managing the access to tape resources**
**may or may not have a disk cache**
**functionality generic but needs to be****   specialize for specific mass storage systems**
**e.g. HRM-HPSS, HRM-Enstore, ...**
**DRM: for managing disk resources**
**functionality generic but needs to be ****   specialize for specific disk systems**
**e.g. DRM-FileSystem, DRM-DPSS, ...**

## HRM role in the Data Grid architecture

**HRM functionality may include :**
**     ****-- queuing of file transfer requests  (kind of a reservation)**
**     ****-- reordering of requests to optimize PFTP order**
**        ****(ordered by files on the same tape)**
**     ****-- Monitoring progress and error messages**
**     ****-- rescheduling transfers that failed**
**DRM functionality may include:**
**     ****-- keeping tracks of files in cache**
**     ****-- managing space reservations**
**     ****-- making decisions on which files to remove when space is needed**
**     ****-- optimizing cache use - sharing files requested by multiple clients**
**     ****-- enforce local policy for cache use**

**HRM role in the Data Grid architecture**

## HRMs & DRMs in a Grid Architecture

**HRMs & DRMs in a Grid Architecture**

- tape system

- client

- Request
- Manager

- HRM

- Disk
- Cache

- client

- Request
- Manager

- DRM

- Disk
- Cache

- Site 2

- Site 1

- Advantages:
- 1.  Each SRM has global view
-      of all requests to it.
- 2.  SRMs  support reservation
-      requests for space and
-      file transfers
- 3.  Can develop uniform error
-      handling codes for all resources
- 4.  Can ask SRMs to queue and
-      monitor requests.
- 5.  Main implication: need APIs
-      to invoke SRMs, and a way for
-      the Grid to pass requests
-      to them.

- DRM

## HRM API description

**API Functionality**
**Request to **_**transfer**_** a file to destination disk**
**A blocking call**
**Request to **_**stage**_** a file to HRM disk**
**A non-blocking call when HRM disk exists**
**Request **_**status**_**/time_estimate**
**How long before file request will be processed**
**Request to **_**abort**_** a file transfer or stage**
**In case file no longer needed**
_**Release**_** a file **
**After file was moved to destination**
**optional to improve system efficiency**
**Call back when file is **_**staged**_

## Examples of call sequences

**actual transfer**

**wait for turn**

**Release**
**file**

**transfer **
**request**

**Time**

**Actual transfer**

**Status,**
**turn **
**reached**

**stage **
**request**

**Time**

**turn **
**reached**

**transfer **
**request**

**status**

**status**

**status**

**status**

**actual transfer**

**transfer **
**request**

**Time**

**turn **
**reached**

**stage **
**request**

**status**

**status**

**status**

**Release**
**file**

**Release**
**file**

**abort**

**abort**

## CORBA IDL for HRM

## HRM-HPSS system functionality

**All transfers go through HRM disk**
**reasons: flexibility of pre-staging **
**disk is sufficiently cheap for a large cache**
**opportunity to optimize for same file requests**
**Functionality**
**queuing file transfers**
**file queue management**
**File clustering parameter**
**Transfer rate estimation**
**Query estimation - total time**
**Error handling**

## SRB Client calls

**Sget**
**blocking call (wait till file is transferred)**
**should only be issued if space on client disk is allocated**
**Sstage**
**non-blocking call (returns time estimate)**
**requires a server if want to be notified**
**can issue status to find out if file was cached**
**Sstatus**
**returns time estimate for file to be staged to HRM’s disk**
**Sabort**
**cancel this file request**

## PPDG Grid Architecture

**PPDG Grid Architecture**

- tape system

- client

- Request
- Manager

- HRM

- Disk
- Cache

- client

- Request
- Manager

- DRM

- Disk
- Cache

- Site 2

- Site 1

- DRM

- Event-file
-  index

- Logical
- query

- Replica
- catalog

- Network
- Weather
- Service

- Event-file
-  index

- Replica
- catalog

- Network
- Weather
- Service

- File
- transfer
- requests

- Logical
- query

- File
- transfer
- requests

## Current PPDG test setup

**Current PPDG test setup**

- Query
- Estimator
- (QE)

- Hierarchical
- Resource
- Manager
- (HRM)

-  **U of Wisc**

- Stage file
- (PFTP)

- File
- Catalog
- (FC)

- Bit-
- Sliced
- index

- Disk Cache

- file
- caching

- Client

- Sget,
- stage,
- sstatus,
- sabort

- Purge file

-  Set of files

-  **LBNL**

-  **LBNL**

- Disk Cache

-  Logical
- Query

** ****CORBA**

**Storage Resource Broker**
**(SRB)       **

- _Note_: there is no
- request manager in this figure.
- the _Client _ is
- acting as its own the
- _Request Manager_

## SRB-HRM interaction

- Hierarchical Resource Manager (HRM)

- esrb.server

- Wisc
- Client 2

- Wisc
- Client 3

- fork

- IPC

- SRB Server
- @Wisc

- Wisc
- Client 1

- SRB child process @LBL

- fork

- fork

- esrb.client

- esrb.client

- IPC

- IPC

- SRB child process @LBL

- SRB child process @LBL

- SRB Server
- @LBL

- esrb.client

- CORBA

- CORBA

- CORBA

- CORBA

- HRM

- esrb.server

- SRB Child @LBL

- esrb.client

- Write to FIFO:
- /tmp/lblsrb.512

- fork

- FID: 512

- 1. FID 512
- 2. Call stage

- It has a table to store FID and requested file Name

- When stage is done
- Write to FIFO:
- /tmp/lblsrb.512

## Status

**Functionality that was tested**
**request a file transfer - Sget**
**A single Client**
**Functionality that exists but was not tested**
**Sget, Sstage, Sstatus**
**multiple Clients concurrent requests**
**use of replica catalog developed by SDSC**
**Functionality that does not exist**
**A request manager that uses the replica catalog**
**A request manager that accesses multiple sites**
**A request manager that makes informed choices**
**A disk resource manager**

## Optional Slides

**The following slides shows how HRM****implements the functionality it provides**** **
**They will be shown only if time permits**

## Queuing File Transfers

**Number of PFTPs to HPSS are limited**
**limit set by a parameter - No_PFTP**
**parameter can be changed dynamically**
**HRM is multi-threaded**
**issues and monitors multiple PFTPs in parallel**
**All requests beyond PFTP limit are queued**
**File Catalog used to provide for each file**
**HPSS path/file_name**
**Disk cache path/file_name**
**File size**
**tape ID**

## File Queue Management

**Goal**
** ****minimize tape mounts**
** ****still respect the order of requests**
** ****do not postpone unpopular tapes forever**
**File clustering parameter - FCP**
**If the file at top of queue is in Tape****i ****and FCP > 1 (e.g. 4) then up to 4 files ****from Tape****i**** will be selected to be ****transferred next**
**then, go back to file at top of queue**
**Parameter can be set dynamically**

**F****1****(T****i****)**

**F****3****(T****i****)**

**F****2****(T****i****)**

**F****4****(T****i****)**

**1**

**2**

**3**

**4**

**5**

**Order**
**of file**
**service**

## File Caching Order fordifferent File Clustering Parameters

**File Clustering Parameter = 1 **

**File Clustering Parameter = 10**

## Transfer Rate (Tr) Estimates

**Need Tr to estimate total time of a query**
**Tr is average over recent file transfers from the****time PFTP request is made to the time transfer completes.  This includes:**
**mount time, seek time, read to HPSS Raid,****transfer to local cache over network**
**For dynamic network speed estimate**
**check total bytes for all file being transferred****over small intervals (e.g. 15 sec)**
**calculate moving average over n intervals****(e.g. 10 intervals)**

## Query Estimate

**Given: transfer rate Tr.**
**Given a query for which:**
**X files are in cache**
**Y files are in the queue**
**Z files are not scheduled yet**
**Let s(file_set) be the total byte size of ****all files in file_set**
**If Z = 0, then**
**QuEst = s(Y)/Tr**
**If Z = 0, then**
**QuEst = (s(T)+ q.s(Z))/Tr****where q is the number of active queries**

**F****1****(Y)**

**F****3****(Y)**

**F****2****(Y)**

**F****4****(Y)**

**Y**

**T**

## Reason for q.s(Z)

**20 Queries of length ~20 minutes**
**launched 20 minutes apart**

**20 Queries of length ~20 minutes**
**launched 5 minutes apart**

**Estimate pretty close**

**Estimate bad - request**
**accumulate in queue**

## Error Handling

**5 generic errors**
**file not found**
**return error to caller**
**limit PFTP reached**
**can’t login**
**re-queue request, try later (1-2 min)**
**HPSS error (I/O, device busy)**
**remove part of file from cache, re-queue**
**try n times (e.g. 3), then return error****“transfer_failed”**
**HPSS down**
**re-queue request, try repeatedly till successful**
**respond to File_status request with “HPSS_down”**

## File Tracking (1)

**HPSS**
**down**

**HPSS**
**down**

## File Tracking (2)

## Dynamic Display of Various Measurements